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Abstract 


In the context of binary program analysis, input crafting describes the task of 
finding an input that directs the control flow from the user input to a defined 
location in the program. One known strategy to automatise input crafting is 
symbolic execution, which operates symbolically on program paths and derives 
path constraints that are solved by an SMT solver. This work describes a novel 
approach to input crafting, based on techniques of bounded model checking. The 
main idea is to unroll the control flow graph of a function up to a certain bound 
and transform it into a formula of first-order logic. An SMT solver decides the 
input crafting problem on the basis of this formula. For this purpose, data flow 
analysis and path selection are encoded as a logical problem. In this dissertation, 
the process of bounded model checking for input crafting on the binary level 
are derived and its feasibility, as well as its efficiency, are evaluated. Therefore, 
an architecture-independent framework for bounded model checking on the 
binary level has been implemented. The results show that this approach works 
efficiently for different architectures and for real-world binaries, such as WPA 
supplicant. In addition, it is demonstrated that SMT solvers exploit the structure 
of programs and solve the path selection problem with great efficiency. The 
main problems of this approach are the necessity to precisely model the program 
semantics, as well as a lesser degree of efficiency for nested memory writes. As a 
consequence, API functions must be included in the program semantics; function 
calls decrease the efficiency. Finally, improvements on the implementation level 
and further research on the basis of bounded model checking are discussed in 
the context of application security. 
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1 Introduction 


This chapter introduces bounded model checking as a means to craft inputs for 
binary programs. After presenting the motivation, the related work will be discussed. 
Then, the research question as well as the contributions of this dissertation will be 
formulated. Finally, the dissertation structure will be outlined. 


1.1 Motivation 

Crafting inputs for binary programs is a common task in the field of reverse engineering 
and application security. In the area of software piracy, input crafting will be used 
for keygenning, in which a valid serial number is generated to activate a software 
application (cf. Section 11 0 )- In cryptography, a cryptosystem is insecure if it is 
possible to craft the key for a known plaintext /ciphertext pair (cf. Section 1.13.1 !)■ 
In the context of exploit development, input generation will be utilised to determine 
whether vulnerable code can be reached by user input |3 . 

In the last decade, methods of static program analysis and software verification 
were used to automatise this task in a broader context. Data flow analysis (cf. Sec- 
tion 9.2 [^), symbolic execution (5] and constraint analysis j6] are three of those 
methods that are frequently utilised @01- In the context of input crafting, the 
first method acquires information about the flow of user input in a control flow graph. 
Then, only the paths that are affected by the user input have to be considered. In 
dependence on the symbolic user input, the second method derives a set of logical 
constraints of a program path. These constraints will be resolved by a solver for 
satisfiability modulo theories (SMT) 1 9 . If the constraints can be satisfied, a valid 
user input will be returned. 

The main limitation of these techniques is their hardness, in the general case. For 
instance, the satisfiability modulo theories are undecidable [10| and, in terms of 
symbolic execution, the amount of paths grows exponentially in the number of 
branches 11 . Nevertheless, heuristics for path selection [7| and the ability of 
SMT solvers to exploit the structure of problems [12] allow the application of those 
techniques on real-world programs. 

Bounded model checking 1 13 is a technique that has not much gained attention 
in the contexts of reverse engineering or binary application security. In contrast 
to symbolic execution, which creates a logical formula of a path, bounded model 
checking generates a logical formula consisting of a control flow graph, which will be 
unrolled up to a certain bound, and properties that have to hold. This formula will 
be solved by an SMT solver, which will, if possible, return a satisfying assignment. 
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1.2. RELATED WORK 


Since SMT solvers exploit the structure of problems, they may operate efficiently 
on data flow analysis and path selection problems. Combined with bounded model 
checking, the SMT solver itself may decide the input crafting problem. The goal of 
this dissertation is to determine the feasibility and efficiency of input crafting for 
binary programs, based on bounded model checking. 


1.2 Related work 


Symbolic execution was introduced by Boyer, Elspas and Levitt |l4] in 1975 and 
has been used for program testing and program proving. In the context of program 
testing, symbolic execution has been utilised for input crafting, known as test case 
generation [7J [L4j 15, .16] |l7]. In 2008, a state-of-the-art symbolic execution engine, 
KLEE [7 , was introduced, which operates on the intermediate presentation of 
LLVM 118 1 and presents heuristics for efficient path selection. 

It is well-known that SMT solving is undecidable in general case and that the 
satisfiability problem is NP-complete (To] . In recent years, active research on efficient 
SMT solvers for various applied problems has been done 1 19 

As a consequence, symbolic execution and SMT solving will be used frequently in the 
context of application security. For instance, AEG 0 (2011), Mayhem [22] (2012) 
and SAGE [8] (2012) apply those techniques for automated exploit generation, both 
static and dynamic, as well as on source code and the binary level. FuzzBALL 1 23 
(2011) utilises symbolic execution and SMT solving in the area of fuzzing. In addition, 
Firmalice 1 24 1 (2015), whose binary analysis platform has been released as angr 1 25] , 
detects vulnerabilities in firmware. Finally, Vanegue, Heelan and Rolles 1 12 (2012) 
proved the power of SMT solvers for program analysis in the context of vulnerability 
checking, exploit generation, input crafting and cryptanalysis, amongst others. 


20 21 


In the held of hardware verification on the basis of SAT solvers, bounded model 
checking 1 26 1 (1999) has been derived from model checking [27] (1982) and, especially, 
from symbolic model checking 1 28 (1992). Armando, Mantovani and Platania |l3 
(2006) introduced bounded model checking on the basis of SMT solvers for software. 
Bounded model checking has also been used to verify C code [29] (2004). On the 
assembly level, it has been applied to the verification of embedded C programs 30 
(2005), concurrent programs j3l] (2005) and software for microcontrollers (2010 32 
and 2014 [33 ). When applied to security, bounded model checking has been used 
for verifying and analysing security protocols (2004 |34| and 2008 [35 ), web applica- 
tions [36 1 (2004) and security properties of control how graphs [37] (2001). Sinz, Falke 
and Merz [38] (2010) introduced the LLBMC (low-level bounded model checker) [39], 
which operates on the intermediate representation of LLVM and locates bugs as 
illegal memory access, integer overflows and others in C programs. However, bounded 
model checking has not been applied to binary application security by now. 



1.3. RESEARCH QUESTION 
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1.3 Research question 

To close the gap in research presented earlier, this dissertation examines whether 
the application of bounded model checking is feasible and efficient to craft inputs for 
binary programs. To provide an answer to this question, a process to apply bounded 
model checking to binary programs must be developed first. The main problem 
thereby lies in the logical encoding of the control and data flow. In addition, input 
crafting has to be expressed as a problem of bounded model checking. Furthermore, 
the research question inquires about the strengths and weaknesses as well as the 
limits of this approach. 


1.4 Contributions 

The contributions of this dissertation are twofold. First, a novel approach to craft 
inputs for binary programs, on the basis of bounded model checking, will be presented. 
The efficiency as well as the strengths and weaknesses of this approach will be 
discussed. Second, Cylyx, a cross-platform and architecture-independent binary 
analysis framework for bounded model checking will be introduced. Cylyx is the first 
publicly existing framework for bounded model checking on the binary level and it 
forms the basis for other use cases, such as vulnerability detection or deobfuscation. 


1.5 Dissertation structure 

Chapter [2] lays the theoretic foundations required for binary input crafting on the 
basis of bounded model checking. It describes graph-theoretic concepts, formally 
defines the input crafting problem and outlines the main ideas of symbolic execution 
and data flow analysis. Additionally, the theory and terminology behind SMT 
solvers and bounded model checking is introduced, as well as the binary analysis 
framework, Miasm, and its intermediate representation, Miasm IR. On the basis of 
this theory, the process of bounded model checking to craft inputs for binary programs 
is discussed in detail in Chapter [3] the general idea is derived, the assumptions and 
prerequisites are mentioned, the main steps - graph transformations, rewriting the 
intermediate representation and static single assignment - highlighted and, finally, the 
memory model, as well as the procedure of formula generation, explained. After that, 
Chapter [4] evaluates the introduced approach and answers the research question. To 
this end, tests will be applied in five case studies in order to scrutinise the advantages, 
disadvantages and robustness of the approach. Lastly, Chapter [5] summarises the 
results and indicates future work. 



2 Foundations and concepts 


This chapter lays the theoretical groundwork for input crafting via bounded model 
checking. For this purpose, graph-theoretic concepts will first be introduced, which 
facilitate the analysis of the control flow of assembly code. Subsequently, the input 
crafting problem will be defined. Then, symbolic execution - a common technique 
used for input crafting - will be discussed. Building on that, the main concepts that 
will be used in this work for input crafting are to be described; namely, they are 
static data flow analysis and constraint solving. For this, ideas of data flow analysis 
will be explained. Following this, satisfiability modulo theories, those behind the 
constraint solving method, will be elucidated. After clarifying the logical background, 
bounded model checking, a technique based on said theory, will be presented. Lastly, 
a brief introduction to the Miasm framework will be given, with special attention 
paid to its intermediate representation. 


2.1 Concepts of graph theory 

In this section, graph-theoretic concepts, with the focus on analysis of control flow 
graphs, will be explained. First, basic terminology of graph theory will be defined. 
Second, control flow graphs will be introduced. Third, loops in control flow graphs 
will be discussed. To provide consistency and simplicity, all definitions in this section 
will be adapted from standard definitions found in the literature of graph theory and 
compiler construction j4] |40 , 4dj . 

2.1.1 Basic terminology 

A directed graph is a data structure which connects a set of nodes by arrows called 
oriented edges. An oriented edge from a node a to a node b indicates a relation from 
a to b (cf. Section 1.2 1 40 j ) . 

Definition 2.1.1 (directed graph). A directed graph or digraph is a pair G = 
( V,E ), where V is a non-empty finite set whose elements are called vertices or nodes 
and E is a finite set of ordered pairs of distinct vertices, which are called arcs or 
edges. 

For instance, let G = (V, E) be a directed graph. Then, the set of vertices will be 
defined as V = {a, b, c, d, e, /, g} and the set of edges E = {(a, b), (a, c), [b, d), (6, e), 
(d, /), (e, /), (e, b ), ( g , d)}. The directed graph is illustrated in Figure [2d~j 

Hence, terminology describing traversals on directed graphs can be defined. Fol- 
lowing, the terms walk, path and cycle will be distinguished (cf. Section 1.4 (40 ]). 
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2.1. CONCEPTS OF GRAPH THEORY 



Figure 2.1: Directed graph with two heads and one cycle 


Additionally, the concepts of predecessors and successors of nodes will be defined 
(cf. Section 1.2 40 and Section 1.5 40|). In both cases, the definitions are adapted 


from the literature. 


Definition 2.1.2 (walk, path, cycle). A (directed) walk in a directed graph G is 
a sequence of vertices (vq, v\, . . . , v n ) along the edges (vq, v\), (ui, ^ 2 ), • • ■ , (v n -i,v n ). 
A walk of distinct Vi is said to be a (directed) path. A (directed) cycle is a walk where 
vq = v n and where {v\,V 2 , ■ • • , v n -{\ are distinct. 

Definition 2.1.3 (predecessor, successor, reachable, head, tail). For an edge 
e = ( Vi,Vj ) in a directed graph G = (V,E), Vi will be called the direct predecessor 
of Vj and Vj the direct successor of m. If there exists a walk from a vertex Vi to 
a vertex Vj and (vj,Vk) £ E, then ig will be said to be a predecessor of Vk and Vk 
to be a successor of m. The set of predecessors of a vertex v will be referred to as 
PRED(u); the set of successors of a vertex v will be referred to as SUCC(u). Equally, 
the set of direct predecessors of a vertex v will be denoted as DPRED(u) and the set 
of direct successors DSUCC(u). If a vertex Vj is a successor of a vertex Vi, then Vj 
is reachable from v l . Additionally, a vertex is reachable from itself. A vertex with no 
predecessors will be called head or root, a vertex with no successors will be called tail 
or leaf. 


Regarding Figure 2.1 (a, 6, e, b , d, f) is a walk, but not a path; (a, b, e, /) is a path. 
(6, e) is a cycle with no nodes between start and end. The sets of predecessors of 
c and / are PRED(c) = {a} and PRED(/) = {a,b,d,e, g}\ the direct predecessors 
of / are d and a, DPRED(/) = {a, d}. The nodes a and g are heads of the graph 
because they do not have any predecessors; c and / are tails of the graph because 
they do not have any successors. There exists a path from a to c, but not from g to 
c. Therefore, c is reachable from a, but not from g. 


2.1. CONCEPTS OF GRAPH THEORY 
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Informally, a part of a directed graph can also be a directed graph. A subgraph of a 
directed graph is a graph whose vertices and edges are subsets of the sets of vertices 
and edges of the directed graph (cf. Section 1.2 1 40 : ) . 

Definition 2.1.4 (subgraph). A directed graph G' = {V',E') is said to be a 
subgraph of a directed graph G = (V, E), denoted as G' C G, if the following holds: 

1. V' C V. 

2. E' C E. 

3. Ve = (v±,V 2 ) E E' : v\ E V' A V 2 E V' . 


The third characteristic means that every node which is part of an edge in the set 
of edges must also be in the set of nodes. For instance, let the set of vertices be 
V 1 = {a,b,c} and let the set of edges be E' = {(a, 6), (a, c)}. Then, G' = ( V',E ') 
is a subgraph of G, the graph in Figure |2.1| However, let the set of vertices be 
E* = {(a,b),(b,d)}. In that case, G* = (V',E*) is not a subgraph of G because 
d 0 V'. 

As of now, strongly connected components can be introduced. In a strongly connected 
component, every node is reachable from every other node (cf. Section 1.5 |40 ) |42 . 

Definition 2.1.5 (strongly connected component (SCC)). A strongly connec- 
ted component or SCC of a directed graph G = (V, E) is a subgraph G' = (V 7 , E') of 
G in which paths from Vi to Vj and from Vj to Vi exist Mvi , Vj E V 1 with Vi ^ Vj . A 
strongly connected component is maximal if no vertices or edges from G can be added 
to G 1 while remaining strongly connected. 


To illustrate, each of the nodes a, c, d, f and g from the graph in Figure |2.1| is 
strongly connected to itself, while G' = (V 1 , E'), with V' = { b , e} and E' = {(6, e), 
(e, &)}, is a strongly connected component between b and e. Since no other edge or 
node can be included, these strongly connected components are maximal. 


On the basis of Definition |2.1.1[ in a directed graph, an edge is a pair of distinct 
nodes. This property implies that an edge from a node to itself cannot exist; self-loops 
are not possible. (However, this exclusion does not restrict handling of self-loops in 
control flow graphs, as Section 2.1.5 will show.) As a result, a directed acyclic graph 
can be defined as follows 143 1 . 


Definition 2.1.6 (directed acyclic graph (DAG)). A directed acyclic graph or 
DAG G = (V, E) is a directed graph with no directed cycles. The number of vertices 
|V| = n is equal to the number of strongly connected components in G. In other 
words, each vertex in G is only strongly connected to itself. 


The graph G illustrated in Figure |2.1| is not acyclic because the nodes b and e 
are strongly connected. The subgraph G’ = { V',E '), with V = {d,f,g} and 
E' = {(< 7 , d), (d, /)}, is a directed acyclic graph since every node d, f and g is only 
strongly connected to itself. 
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2.1. CONCEPTS OF GRAPH THEORY 


2.1.2 Depth-first traversal 


After the basic terminology required for control flow analysis has been defined, one 
algorithmic concept called depth- first search (DFS) or depth- first traversal, which is 
the fundament for a multitude of graph analysis algorithms [43,j41, 42, 44, 45 , will be 
introduced. DFS is an algorithm that explores a directed graph, starting from a given 
node v, until every node that is reachable from v has been visited (cf. Section 4.1 1 40 j ) . 


The iterative algorithm in Algorithm 2.1.1 has the complexity 0(|F| + l-El). Given a 
node v, DFS marks the node as visited and pushes the direct successors of v onto a 
stack. Succeeding, v will be set to the last element on the stack and the previous 
step will be repeated. If the stack is empty, all reachable nodes from the given node 
v have been visited. The result is a list of nodes sorted by the order in which they 
have been visited. 


Algorithm 2.1.1: Depth- first search (DFS) 


Data: directed graph G = (V,E), start node v £ V 
Result: list of nodes T = [no, v \, . . . , v n ] 

1 S <— EmptyStack 

2 Del 

3 T i — EmptyList 

4 PUSH(S, v) 

5 while S EmptyStack do 

6 v <- POP(S) 

7 if v 0 D then 

/* mark v as visited 
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D <— DU {u} 

/* add v to the ordered list of visited nodes 
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10 

11 

12 

13 

14 


APPEND (T,v) 

/* push the direct successors of v onto the stack 
foreach s £ DSUCC(u) do 
PUSH(S,s) 

end 

end 

end 


*/ 

*/ 

*/ 


For the graph in Figure 2.1 DFS(a) = [a, c, b, e, /, d] is one possible DFS traversal, 
starting at node a; DFS(a) = [a, b, d, f, e, c] is also possible. The traversal DFS(a) = 
[a, b, c, e, /, d] is impossible because after b has been visited, one of its direct successors, 
d or e, as well as all the nodes reachable from those have to be visited before c can 
be visited. 


2.1. CONCEPTS OF GRAPH THEORY 
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2.1.3 Control flow graphs 


Since the graph-theoretic terms and their characteristics have been defined, they 
will be adapted and extended in the context of compiler construction |4] and static 
program analysis 46 . Working on a sequence of program statements, the construction 


of control flow graphs based on those statements facilitates analysis of control and 
data flow; therefore, it supports reasoning about the behaviour of the sequence 
of program statements. Thereby, an order of execution of a sequence of program 
statements will be informally defined as control flow. 


Definition 2.1.7 (control flow). The control flow of a sequence of program state- 
ments is the order in which the statements will he executed or evaluated. 


A sequence of program statements can be partitioned into basic blocks. A program 
statement is an entry of a basic block if that statement is at the beginning of 
a sequence of program statements, if it is a target of a jump statement or if it 
immediately follows a jump statement (cf. Section 8.4.1 |4]). Building on that, a 
control flow graph is a directed graph with basic blocks as nodes and two additional 
nodes, Entry and Exit [4jLj. 

Definition 2.1.8 (basic block, control flow graph (CFG)). A basic block is 
a sequence of program statements that is entered at the first and exited at the last 
statement. A control flow graph or CFG is a directed graph where nodes represent 
basic blocks and edges represent transfer of control between basic blocks. Besides 
the basic blocks, a control flow graph has two additional nodes called Entry and 
Exit. There is an edge from Entry to any basic block where statements can enter 
the sequence of program statements; there is an edge from any basic block where 
statements can leave the sequence of program statements to Exit. 


Defining a control flow graph with an Entry and an Exit node is a simplification 
without loss of generality. If a sequence of program statements has several ending 
points, the associated basic blocks will have edges to the node called Exit ; the same 
holds for several starting points. For instance, if only one certain starting point s 
is of interest, then the edges from the node Entry to the other basic blocks can be 
removed (excluded the edges to the basic block that is associated with s). 


In the control flow graph illustrated in Figure 2.2, Entry has one outgoing edge 
to a and Exit one incoming edge from /. If the control flow graph represents a 
sequence of program statements defining a function, then a can be interpreted as 
the start and / as the end of the function. For the purposes of illustration, in this 
example, the basic blocks contain labels instead of program statements. This serves 
as a visualisation for the definitions in the next sections. 


Since a control flow graph is a graph of basic blocks and a basic block is a sequence 
of program statements, a control flow graph represents the control flow of program 
statements. A basic block b with two direct successors requires a conditional statement 


10 


2.1. CONCEPTS OF GRAPH THEORY 



Figure 2.2: Control flow graph with one reducible and one irreducible loop 


that defines whether the control flow will be directed to the one or the other direct 
successor of b. This condition will be called jump condition. 

Definition 2.1.9 (jump condition). Let G = (V, E) be a control flow graph. If 
a basic block b G V has two direct successors s\,S 2 £ V, then the last conditional 
program statement in b will be said to be the jump condition of b. Depending on the 
jump condition j , the conditional statement j 7 s\ : S 2 directs the control flow to si , 
if j is true, otherwise to S 2 - 


A jump condition can be either true or false. It may depend on complex statements; 
for instance, x + y + z == x ■ y as well as x < y can be jump conditions. The last 
program statements in the basic blocks a, b, c, d, e and h in Figure [272] are conditional 
statements which rely on a jump condition. For example, let the jump condition for 
b be x < y. Then, the conditional statement could be x < y ? d : f; if x < y is true, 
then the control flow will be directed to d, otherwise to /. 


Lastly, the concept of a walk or path can be adapted to control flow graphs. An 
execution path will be defined as a path that directs the control flow from Entry to 
a basic block b in a control flow graph. In contrast to Definition 2.1.2 an execution 
path will be defined as a walk in a control flow graph; an executed path may be 
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cyclic. Additionally, several execution paths may exist between two basic blocks. 
However, there exists only one execution path at a specific point in time, since an 
execution path represents an instance in which a sequence of program statements is 
evaluated. 

Definition 2.1.10 (execution path). Let G = (V,E) be a control flow graph. An 
execution path p is a walk in the control flow graph that starts in Entry and directs 
the control flow in the sequence of program statements represented in the control flow 
graph along p. If there are different possible execution paths, only one of them can be 
taken at a specific point in time. 

In the illustrated control flow graph, [Entry, a, c, e, /] and [Entry, a, c, e, h, e, h, e, h, f] 
are two possible execution paths. It may be noted that there exists a cycle between 
e and h. 


2.1.4 Dominance relations 


In a control flow graph, the dominance relations between nodes are a strong property 
which enables techniques such as optimisation (cf. Section 10.4 |4j) or loop analysis 
(cf. Section 2.1.5). The dominance relations are also important for static single 

The definitions in this section 

|4ll. 


assignment, which will be introduced in Section 2.1.6 
are based on the work by Cytron et al. 


Definition 2.1.11 (domination, dominator, strict dominator). Let G be a 

control flow graph. A node Vi dominates a node Vj, if every path from Entry to Vj 
passes through Uj. Vi is called the dominator of Vj. If a node Vi dominates a node 
Vj and Vi Vj, then Vi strictly dominates v y The set of dominators of a node v 
will be denoted as DOM(u). If a node v t dominates a node Vj, it will be denoted as 
v\ domu 2 ,‘ if a node Vi strictly dominates a node Vj, it will be denoted as v\ sdomr> 2 . 


For instance, in Figure 2.2 it is DOM(6) = {Entry, a, 6}, whereby b is dominated by 
itself, but strictly dominated by Entry and a. As well, / is dominated by itself and 
strictly dominated by Entry and a, DOM(/) = {Entry, a, /}; Exit is dominated by 
itself and strictly dominated by Entry, a and /, DOM(Exit) = {Entry, a, f, Exit}. 
Additionally, the dominators of h are Entry, a, c and h. 


The closest strict dominator of a node v will be called the immediate dominator. 
Each node besides Entry has precisely one immediate dominator. 


Definition 2.1.12 (immediate dominator). Let G be a control flow graph. Vi 
is the immediate dominator of Vj if Vi is the nearest (direct) predecessor of Vj that 
strictly dominates Vj on all paths from Entry to Vj. The immediate dominator of a 
node v will be denoted as IDOM(w). 


To exemplify, it holds that IDOM(o) = Entry, IDOM(/) = a and IDOM(/i) = c. 
The only node that does not have an immediate dominator is Entry, because Entry 
does not have any strict dominator. 
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The dominator tree of a control flow graph is a data structure that represents 
dominance relations in a graph. If there exists an edge from a node u to a node v in 
the dominator tree, then u is the immediate dominator of v |41| . 

Definition 2.1.13 (dominator tree). Let G = (V,E) be a control flow graph and 
E' = {(IDOM(u), u)|u € V\{Entry}} . Then G' = (V,E') is called the dominator 
tree of G. 

As stated above, Entry, a and / strictly dominate Exit, whereby IDOM(Eaht) = /, 
IDOM(/) = a and IDOM(a) = Entry. Therefore, [Entry, a, f, Exit \ is a path in the 
dominator tree, as depicted in Figure |2.3| Throughout the course of the dissertation, 
the terms predecessor and successor will refer to control flow graphs; the terms parent 
or ancestor and child or descendant will refer to dominator trees. 



If a node Vj in a control flow graph G has more than one incoming edge, then there 
exists a path convergence of at least two paths in Vj. Let p\ and p 2 be two paths, 
which converge first in Vj . Assume that p± is a path from Entry to Exit, p 2 is a path 
from a node Vi to Exit and Vi 0 p\. Then, it will be said that Vj is in the dominance 
frontier of Uj. 

Definition 2.1.14 (dominance frontier (DF)). A node vj in a control flow 
graph G is in the dominance frontier or DF of a node Vi if Vi dominates a pre- 
decessor of Vj and if Vi does not strictly dominate vj. The dominance frontier 
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of a node Vi will be denoted as DF(uj), whereby it will be formally defined as 
DF(uj) = {vj\3p € PRED(uj) : Ujdomp A — sdomuj)}. 

In Figure [ 2 Y| c dominates the set {c, e, h}. There is a path p\ = [Entry, a, b, f, Exit], 
with / G pi and c 0 p\. There also exists path p-2 = [Entry, a, c, e, f, Exit], with 
cGp2 and f € P2- / is also the first node, where p\ and p2 converge. In other words, 
c dominates e, a predecessor of /, but c does not strictly dominate /. Therefore, / is 
in the dominance frontier of c. Others examples are DF(d) = {b, /} and DF(g) = {&}. 
Since a dominates every node besides Entry, there cannot be a path p from Entry 
to Exit with a 0 p. Thus, it is DF(a) = 0 . 


2.1.5 Reducible and irreducible loops 


Since a control flow graph represents the control flow of program statements, in- 
formally, a cycle in a control flow graph represents a loop in a sequence of program 
statements. For that reason, within the framework of this dissertation, cycles in a 
control flow graph will be referred to as loops. The set of nodes in a loop will be 
called a loop body. If it is clear from the context, the terms loop and loop body will 
be used interchangeably. In relation to Definition 2 . 1.5 a cycle (loop) in a directed 
graph (control flow graph) is a strongly connected component because each node can 
be visited from any other node within the cycle (loop). In other words, a strongly 
connected component with at least two nodes is an outermost loop in the control 
flow graph 1 42 1 . 


Definition 2.1.15 (loop, loop body). A cycle in a control flow graph will be 
called a loop. The set of nodes within a loop will be called a loop body. 


Definition 2.1.16 (outermost loop). A maximal strongly connected component 
with at least two distinct nodes in a control flow graph G will be said to be an 
outermost loop. 


The term outermost does not reveal anything about the structure of the nodes within 
the strongly connected component. If I2 is an outermost loop, there may exist a loop 
l\ which is a proper subset of I2. h will be called an inner loop, I2 an outer loop. In 
other words, loops can be nested. 

Definition 2.1.17 (nested loops, inner loop, outer loop). Let l±, I2 and I3 be 

loops in a control flow graph G. If it holds that l\ C I2 C I3, then l\ will be called an 
inner loop of 1 2 and I3; I2 will be said to be an inner loop of I3, but an outer loop of 
l\ . The loops l\, I2 and I3 are nested. 

Definition 2.1.18 (innermost loop). A loop in a control flow graph that does not 
contain any further loops will be called the innermost loop. 

For instance, the control flow graph G in Figure | 2 . 2 | has two outermost loops - 
l\ = {e, h} and I2 = { 6 , d, g}. Let there be an additional edge ( g , d). Then, I3 = {d, g} 
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is an inner loop and I 2 an outer loop. Additionally, since I 3 does not contain any 
further loop, it is the innermost loop. 

The first node of a loop l that will be visited in a control flow graph G will be called 
a loop header or loop entry. Formally, a loop header is a loop node which has a direct 
predecessor outside of the loop. A loop may have several headers. If it has a single 
entry, the loop will be called reducible , otherwise it will be called irreducible |42 . 

Definition 2.1.19 (loop header). Let l be a loop in a control flow graph G = (V, E). 
If a node v £ l has a direct predecessor p with p 0 l, then v is a header or entry of l. 
The set of entries of a loop l is defined as H = {u|3p £ DPRED(u) : p 0 l A v £ l}. 

Definition 2.1.20 (reducible loop, irreducible loop). If a loop in a control 
flow graph has a single loop entry, it is a reducible loop; if a loop has multiple loop 
entries, then it is an irreducible loop. 


The two loops in Figure 2.2 are l\ = {6, d, g} and I 2 = {e, h}. The node 6 £ / 1 is the 
only node in l\ that has a direct predecessor outside of l \ . Thus, l\ has a single loop 
entry and is a reducible loop. In I 2 , e and h have a direct predecessor outside of I 2 . 
Therefore, I 2 has two loop headers and is an irreducible loop. 


Since a reducible loop has a single entry point, the loop header dominates all nodes in 
the loop. Thus, there exists an edge from a node of the loop body to the dominator 
of the loop. Such an edge (v, u ) from a node v to a dominator u of v is called a back 
edge. Due to the fact that udomr implies the existence of a path from u to v, a back 
edge (v, u ) implies the existence of a loop, which is dominated by u. The loop body 
consists of u and all the nodes which are able to reach u without walking through v. 
In other words, the loop body consists of all nodes that can be visited by performing 
a depth-first search from u to v on the reverse control flow graph. Such a loop will 
be defined as a natural loop (cf. Section 9.6.6 0)- 


Definition 2.1.21 (back edge). Let G = (V, E ) be a control flow graph. If a node 
Vi dominates a node Vj, Vidomvj, and there exists an edge e = ( Vj,Vi ) £ E, then e 
is said to be a back edge. 


Definition 2.1.22 (natural loop). Let G = (V,E) be a control flow graph and 
e = ( Vj,Vi ) £Ed back edge. Then, Vj is be said to be the loop header of a natural loop, 
whose loop body consists of the set of nodes obtained from a depth-first search from Vj 
to Vi on the reverse control flow graph G' = (V,E > ), with E' = {(e 2 , ei)|(ei, e 2 ) £ E}. 


As stated above, l\ = {b, d,g} is a reducible loop with the loop entry b. Since there 
is an edge from g to b and b is a dominator of g, e = (g, b) is a back edge. The edges 
(b,d), (d,g) and ( g,b ) are part of the loop. Hence, the loop edges of the reverse 
control flow graph are (d, b ), (g, d) and (6, g). Performing a depth-first search from g 
to b on the reverse edges, DFS(g) = [ g , b, d] is one possible result. Thus, the set of 
those nodes build the loop body of the natural loop dominated by b. 
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For the sake of completeness, some properties of natural loops will be mentioned. If 
two natural loops have the same header, one is either properly contained within the 
other or they can be combined to form one loop. If two natural loops have distinct 
headers, they are either disjoint or nested. If a control flow graph only has natural 
loops, it is called reducible (cf. Section 9.6.6 [4]). 

Definition 2.1.23 (reducible control flow graph). A control flow graph is re- 
ducible if all of the loops in that control flow graph are reducible loops. Otherwise, 
the control flow graph is irreducible. 


The properties of reducible loops cannot be conferred to irreducible loops. Since 
there is no single loop entry, the dominance relations in the loop are different. One 
strategy, called node splitting , applies graph transformations such that irreducible 
loops will be transformed into reducible loops. The disadvantage of that approach is 
its exponential running time [45]. 


the requirement that an edge in a directed graph has to be a tuple of distinct nodes 
excludes self-loops, by definition. This means that an edge ( v , v) cannot exist for a 
node v in a control flow graph. Assume, it would be possible and such an edge could 
exist. Additionally, assume that this self-loop would not be part of any other loop. 
Then, this loop would not be detected as an outermost loop because it would only 
be strongly connected to itself. However, a self-loop is a natural loop, since a node 
dominates itself by definition and therefore, a back edge exists. In other words, if a 
control flow graph contains a self-loop, it can be detected. 


Last, as stated in Section [2. 1.1[ in relation to Definition 2.1.5|and Definition 2.1.6 


2.1.6 Static single assignment 

Static single assignment (SSA) is a representation of program statements that has two 
properties. First, each variable assignment is unique in that each use of a variable is 
defined by exactly one assignment. Second, variable assignments on different control 
flow paths can be distinguished in converging paths with ^-functions. The concepts 
described in this section are based on the work by Cytron et al. 1 41 and Alfred V. 
Aho et al. (cf. Section 6.2.4 j4]). To begin, the terms def-use chain or DU chain , 
LHS and RHS have to be defined. 

Definition 2.1.24 (def-use (DU) chain, left-hand side (LHS), right-hand 
side (RHS)). In a sequence of program statements, an assignment of a variable v 
with a variable w has the form v = w. It will be said that v will be assigned to the 
value of w; v will be defined and w will be used. The definition of the assignment 
takes place on the left-hand side or LHS, the use of the assignment on the right-hand 
side or RHS. In other words, a variable v will be defined on the LHS and used 
on the RHS. The definition of a variable v and its further uses will be called the 
definition-use, def-use or DU chain of v. A DU chain of a variable v ends with the 
reassignment of v. 
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In Figure [2~4a] the variable x will be defined in the statements x = a+ 7 and x = a + 6 ; 
it will be used in the statements y = x ■ a, y = x — b and z = x + y. Since x will be 
defined twice on the LHS, there exist two DU chains for the variable x. 


x = a + 7 
y = x ■ a 
y = x — b 
x = a + b 
z = x + y 

(a) Program statements 


xq = a + 7 
Vo = x 0 • a 
yi = x 0 ~b 
x\ = a + b 
z 0 = x t + y\ 

(b) Program statements in SSA form 


Figure 2.4: Program statements and their SSA representation 


After the transformation of a sequence of program statements into SSA form, each 
variable definition of a variable v on the LHS will be unique; each variable use of v 
on the RHS will be the last definition of v. For instance, Figure 2.4b shows the SSA 
transformation of the program statements in Figure |2 .4a For the two definitions of 
x, two new variables, xq and x\, will be created. Similarly, the last definition of x - 
xo or x\ - will be used instead of x on the RHS. 


The advantage of the first property of SSA - each variable assignment is unique - 
is the creation of a link from the beginning to the end in the sequence of program 
statements. This enables, inter alia, optimisations as follows: After assignment, yo 
will never be used on the RHS; instead, y\ will be assigned in the following statement. 
That means that yo has no influence on any statement in the sequence of program 
statements and can be removed. A second example of this advantage is the resolving 
of dependencies. For instance, the variable zq depends on x\ and y \ . Further, y\ 
depends on xo and b, whereby xo depends on a and a constant; x\ depends on a and 
b. In other words, zq can be resolved to zq = x\ + y± = ■ ■ ■ = a + b + a + 7 — b, which 
can be simplified to zq = 2 ■ a + 7. 


Relying only on the first property of SSA, a control flow graph cannot be transformed 
into SSA form. As stated in Section 2.1.3, if a basic block b\ has two outgoing 
edges, the control flow depends on a jump condition; the jump condition affects the 
control flow. The flow of control will be directed either to the one or the other direct 
successor of bi . Two different execution paths of the program statements are possible 
- pi , if the jump condition of b\ is true, and P 2 , if the jump condition is false. Then, 
let v be a variable which will be defined in and modified in at least one of the 
execution paths. If pi and p -2 converge in a basic block &2 and v will be used in 62 , 
then v cannot be assigned explicitly. For that reason, the concept of 4> -functions will 
be introduced. 


Definition 2.1.25 (4>-function). In a control flow graph in SSA form, a <!> -function 
for a variable v is a function that combines all last definitions of v in converging 
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execution paths and returns the definition ofv that corresponds to the current execution 
path. An assignment has the form m = <f>(uo, vi, ■ ■ ■ , v n ), with 0 < n < i. 

In general, ^-functions are an abstract concept that combines the previous definitions 
of a variable for further analysis; they are not specified in a concrete or standardised 
way. Below, it will be illustrated how that concept can be used. 



Figure 2.5: ^-functions in a control flow graph 


Figure 2.5 shows a control flow graph in SSA form. There are variable definitions 
for x, y and 2 . In the first basic block, the last definitions in SSA form are xo, yo 
and zq. There exists a jump condition zo < 10. The left successor redefines x and 
y; the right successor redefines x. Both execution paths converge in a block b. If b 
or a successor of b will use a variable x or y, it cannot be said which is the latest 
variable definition. For that reason, new variables — X 3 and y 2 - will be assigned 
for x and y in 6 ; these will be used in the control flow starting in b. The variables 
are defined with ^-functions; those functions signal that one of the two paths is the 
current execution path. For instance, if the right execution path will be taken, then 
X 3 will be assigned to the value of X 2 and y 2 to the value of yo- 


Another example will be given for an analysis based on SSA and the abstract concept 
of ^-functions. For the control flow graph in Figure [23} it can be said which execution 
path will be taken. Let the control flow be directed to the left successor of the first 
basic block, if the jump condition zo < 10 is true; otherwise, it will be directed to 
the right successor. If X 3 == X 2 is true, then the right execution path will be taken. 
That implies that zq < 10 must be false, zq can be resolved to zq = xo + 1 = 4 + 1 = 5. 
In other words, 5 < 10 cannot be true. Therefore, the left execution path is the only 
possible path; it holds X 3 = x\ and y 2 = yi • 
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2.2 Input crafting problem 


The input crafting problem describes the task of finding an input such that certain 
statements in a sequence of program statements will be executed. In the literature, 
the problem has been phrased in different ways |47 48 . Based on the definitions of 


the previous sections, the input crafting problem can be defined in a graph-theoretic 
manner as the search for a user input that directs the control flow from a basic block 
a to a basic block b in the control flow graph. 


Definition 2.2.1 (input crafting problem). Let G = (V,E) be a control flow 
graph and bi,bj E V be two basic blocks. Additionally, let v be a variable which is 
defined by user input in 6,. The input crafting problem asks if there exists an input 
for v that directs the control flow from hi to bj in the control flow graph. 


2.3 Symbolic execution 


Symbolic execution describes a technique which evaluates program paths in a symbolic 
manner, instead of one that is concrete. Working on a control flow graph, certain 
variables will be defined to be symbolic. Then, the program statements in the basic 
blocks will be calculated symbolically - equalities and inequalities will be generated 
in dependence on the symbolic variables and the semantics of the program statements. 
Those equalities and inequalities will be called constraints 0 

Definition 2.3.1 (constraints). For a sequence of program statements, constraints 
are equalities and inequalities derived from the semantics of the program statements. 


48 


The jump condition of a basic block will also be evaluated symbolically. Let the 
last conditional statement of a basic block b, whose direct successors are si and S 2 
and whose jump condition is j, be j ? s i : S 2 - If the next basic block that will be 
executed symbolically is si, then the constraint will be defined such that j is true. 
Otherwise, j has to be false. In other words, the constraints have to be derived along 
an execution path. The constraints of an execution path are said to be the path 
conditions. To receive a concrete solution for the path conditions, the constraints 
have to be solved; if there exists a solution, the path conditions can be satisfied. 


Constraint solving will be discussed in Section 2.5 


Definition 2.3.2 (path conditions). Let G be a control flow graph. For a sequence 
of program statements, the path conditions or path constraints are the constraints 
derived from an execution path in the control flow graph. 


Symbolic execution operates on execution paths. For instance, if that technique will 
be used for input crafting and there exist several possible execution paths, then the 
possible execution paths will be evaluated iteratively. For each evaluated path, it 
will be checked if the path conditions can be satisfied. If they can be satisfied, then 
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an input is known and the input crafting problem is decided. Since the number of 
possible execution paths grows exponentially with the number of branches, exploring 
every path is infeasible. Such exponential growth will be referred to as path explosion ; 
the problem to decide which path should be explored first is known as path selection 
problem [49 , j5] . 

Definition 2 . 3.3 (path explosion, path selection problem). Let G be a control 
flow graph. The exponential growth of execution paths in the number of branches will 
be called path explosion. The path selection problem asks which direct successor of a 
basic block b should be visited first. 


2.4 Static data flow analysis 


The term data flow analysis describes a set of techniques as acquire information 
about the flow of data in the control flow graph (cf. Section 9.2 [4]). The word 
static refers to the characteristic that the sequence of program statements will not 
be executed; contrary to that, dynamic data flow analysis operates on an execution 
path after a sequence of program statements has been executed [50 . 

Techniques as reaching definitions , live variable analysis , dead variable analysis or 
available expressions pertain to a set of a multitude of techniques in the context of 
data flow analysis (cf. Chapter 2 |51|). Those techniques are beyond the scope of 
this work and will not be covered. Instead, certain ideas of data flow analysis will be 
illustrated. 


Data flow analysis on a control flow graph can be performed on the level of basic 
blocks. For each basic block, it is possible to define inputs for variables that enter the 
basic block and outputs for variables that leave the basic block (cf. Section 9.2.3 0). 
The inputs of a basic block b are the union of outputs of its direct predecessors. 
In contrast, the outputs of b are defined as the union of the variables that will be 
defined in b and of the inputs which have not been modified in b. 


For instance, in Figure 2.5 let the basic block bo be the basic block which is the root 
of the graph, b\ be the left direct successor of bo, 62 be the right direct successor 
and, lastly, 63 be the remaining basic block, bo defines the variables xq, yo and 
zq. Therefore, outo = { xo,yo,zo } is the set of outputs. The inputs of b± and 62 
are equal to the outputs of bo, in\ = m2 = outo- The outputs of bi are defined as 
out\ = {x\,y\, 20} because of the redefinitions of x and y. Equally, out2 = {x2,yo, zo} 
are the outputs of &2- Furthermore, the inputs of 64 are the union of the incoming 
outputs - in4 = out\ U out2 = {x\,X2, yo, Vi, zo}- At last, since x and y have been 
redefined, the outputs of 64 are out 4 = {x3, y2, zo}. The SSA representation already 
includes concepts of data flow analysis. For instance, the 4>-f unctions in 64 are 
defined to represent the last variable definition of an execution path, as stated in 
Section 12.1.61 
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2.5 Satisfiability modulo theories 

In this section, the theoretical and practical foundations for constraint solving on the 
basis of SMT solvers will be acquired. For that, the basic principles of first-order logic 
have to be mentioned. On the other hand, handling these principles continuously 
in a formal manner is beyond the scope of this work and unnecessary. Therefore, 
the following sections will laid out as informally as possible, while still retaining the 
required level of formality. 

First, basic terminology will be presented. Then, the functionality of SAT solvers 
will be described such that the power of SMT solvers can be demonstrated. Finally, 
two common theories will be described - the theory of bit vectors and the theory of 
arrays. 


2.5.1 Preliminaries 

The first-order logic (FOL) consists of a set of variables, logical symbols (for instance, 
A, V, -i, 3 and V), nonlogical symbols (e.g., functions) and a syntax, which defines 
how formulas are constructed. The first-order logic without quantifiers (V and 3) 
and nonlogical symbols will be referred to as propositional logic. A formula is a 
well-formed sentence of first-order logic, with respect to the syntax. It is satisfiable 
if there exists an interpretation for which the formula is true; otherwise, it is 
unsatisfiable (cf. Section 1.4 [19] ) . In the context of this work, the word model will 
be used instead of ‘interpretation’. Additionally, in the context of first-order logic, 
if a formula is true, it will be denoted as being equal to 1; if it is false, it will be 
denoted as being equal to 0. 

Definition 2.5.1 (formula, satisfiable, unsatisfiable). A well-formed sentence 
p of first-order logic will be called a formula. If there exists a structure for which a 
formula <p is true, then ip is satisfiable; otherwise, it is unsatisfiable. The structure 
that is satisfiable for <p will be referred to as interpretation or model. 

p = (a V b) A (a V ~>b) is a formula of first-order logic, p = 1 holds for a = 1 and 
b = 0; p is satisfiable and m = [a = 1, b = 0] is a model that satisfies p. 

The simplest formulas of first-order logic are known as atoms or predicates. The term 
literal refers to a predicate or its negation. Predicates that are connected through 
logical symbols can be represented, amongst others, in conjunctive normal form and 
in disjunctive normal form (DNF). (cf. Section 1.3 |19|). 

Definition 2.5.2 (conjunctive normal form (CNF), disjunctive normal 
form (DNF)). A formula of first-order logic is in conjunctive normal form or 
CNF, if it is a conjunction of disjunctions. If a formula is a disjunction of conjunc- 
tions, it is in disjunctive normal form or DNF. 
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To exemplify, the formula (a V 6) A (->a) A (a V c) is in conjunctive normal form; 
(a A b) V (-■ a) V (a A c) is in disjunctive normal form. In these formulas, a, b and c 
are the predicates; the literals of a are a and —> a. 


2.5.2 SAT solver 

Before satisfiability modulo theories and SMT solvers can be described, the sat- 
isfiability problem and the function of SAT solvers will be presented. These are 
basic principles for the next section. The satisfiability problem (SAT) describes the 
problem if a formula is satisfiable. 

Definition 2.5.3 (SAT problem). The satisfiability problem SAT asks if a formula 
p of the propositional logic is satisfiable or unsatisfiable. 

In general, SAT is an NP-complete problem (cf. Section 2.5 [l9 ). Nonetheless, SAT 
solvers are known to perform adequately on many real-world problems. 

Definition 2.5.4 (SAT solver). A SAT solver is a program that decides if a 
formula <p of propositional logic is satisfiable or unsatisfiable. If it is satisfiable, it 
returns SAT and a model; otherwise, it returns UNSAT. 

‘Modern SAT solvers can solve many real-life CNF formulas with hundreds of thou- 
sands or even millions of variables in a reasonable amount of time’ (Chap. 2, p. 
27 [19] ) . The reason for this is the way SAT solvers work. SAT solvers based on the 
Davis- Putnam- Logemann- Loveland (DPLL) algorithm are able to learn when they 
run into a conflict. A conflict is an assignment of the variables in the formula ip, 
such that tp becomes false. Then, a SAT solver creates a conflict clause c and tries 
to solve the new formula <p' = ip A c. In a formula in conjunctive normal form, all 
disjunctions have to be true. A SAT solver randomly picks a variable in a disjunction 
and sets it to true or false. Next, the other variables in this disjunction will be set 
such that the disjunction becomes true. A conflict occurs if a disjunction cannot be 
true with the current variable assignments. While performing an exhaustive search 
on binary trees, the ability to learn conflict clauses facilitates a SAT solver to prune 
the search space significantly. In other words, SAT solvers are efficient for use in 
many real-world problems because they are able to exploit the structure of those 
problems (cf. Section 2.2 jl9| ) . 


2.5.3 SMT solver 


Informally, satisfiability modulo theories (SMT) can be described as an extension 
of SAT. An SMT formula ip is an instance of SAT in which the predicates of <p 
are not limited to propositional predicates; instead, they can be predicates over 
(combined) theories, such as the theory of integers (cf. Section 3.3.3 
of reals (cf. Section 3.4.1 [52 ), the theory of arrays (cf. Section 3.6 
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), the theory 
52 ) and the 
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theory of bit vectors (cf. Section 6.1 ED)- In contrast to SAT formulas, real-world 
problems can be modelled more expressive in SMT formulas (cf. Section 1 (53 1). 
The satisfiability modulo theory problem describes the problem if an SMT formula is 
satisfiable (cf. Section 3 [53j ) . 

Definition 2.5.5 (satisfiability modulo theory problem). Let ip be a formula 
of propositional logic and of a theory T. The satisfiability modulo theory problem for 
the theory T asks if there exists a model of T that also satisfies ip. 


As stated in Section 2.5.2, SAT is NP-complete. Satisfiability modulo theories are, 
in the worst case scenario, undecidable. Quantifier-free fragments of theories - the 
fragments of theories without V and 3 - are NP-complete, in many cases. For instance, 
this holds for the theory of bit vectors and the theory of arrays. The quantifier-free 
fragments are sufficient for modelling many real-world problems |l0 . As well as SAT 
solvers, SMT solvers are known to perform adequately on many real-world problems. 


Definition 2.5.6 (SMT solver). An SMT solver for a theory T is a program that 
decides if a formula of propositional logic and of the theory T is satisfiable or 
unsatisfiable. It returns SAT and a model, if it is satisfiable; otherwise it returns 

UN SAT. 


SMT solvers depend on SAT solvers. There are two common techniques how an 
SMT solver interacts with a SAT solver - eager and lazy SMT techniques. The 
eager technique translates the fragments of SMT formulas in non-propositional logic 
into fragments in propositional logic and solves the transformed formula with a SAT 
solver. In the lazy approach, a SAT solver interacts with a theory solver for a theory 
T. If the SAT solver generates a model, it queries the theory solver whether this 
model is also a model for T. If this is not the case, the SAT solver creates - as 
described in Section 2.5.2 - a conflict clause, generates a new formula and re-starts. 


Otherwise, the SAT solver generates a model that is also satisfiable in T. Then, the 
SMT solver returns SAT and a model m. In general, modern SMT solvers are build 
upon the lazy approach (cf. Section 3.2 [53]). 

Many modern SMT solvers support the SMT-LIB standard |54| , which defines a 
common interface for SMT formulas. These formulas can be solved with different 
SMT solvers. In the following sections, the basic ideas of the theory of bit vectors 
and of the theory of arrays will be illustrated informally. In the SMT-LIB standard, 
the theory of quantifier-free bit vectors is known as QF_B V ; the combined theory 
of quantifier- free formulas over the theory of bit vectors and arrays is known as 
QF^ABV (55). 


2.5.4 Theory of bit vectors 

In this section, the basic concept of the theory of bit vectors and their operations 
will be given. The formulas in the theory of bit vectors will be assumed to be 
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quantifier-free and be represented in QF_BV [56 . A fixed-sized bit vector is an array 
of bits of the length l. 

Definition 2.5.7 (bit vector). A bit vector b of the length l is a tuple b = 
(b 0 ,...,bi_ i), with bi € {0,11- 

Arithmetic and bitwise operations can be formally defined in the theory of bit vectors 
(cf. Section 6.1 [19], Section 1.3 [57] and Section 2.4 (58]). Therefore, arithmetic 
operations, such as addition, subtraction, multiplication or division as well as bitwise 
operations as &, |, -i, ®, << and >>, can be applied to bit vectors. Since bit vectors 
have a fixed-size, they may overflow. To exemplify, bit vectors of the length l = 8 
represent the numbers from 0 to 255. 

2.5.5 Theory of arrays 

The theory of arrays models the behaviour of arrays used in programming lan- 
guages (cf. Section 7.1 [19]). Since arrays fulfil certain characteristics, these have 
to be modelled properly. To point out those properties, the theory of arrays will be 
formally introduced. The following definitions are based on the work by Bradley and 
Manna (cf. Section 3.6 (52 ;|) as well as by Falke, Sinz and Merz (59 . 

Definition 2.5.8 (theory of arrays). The theory of arrays has the signature 
Tie '■ {•[•], ■(•<■), =}• a[i\ or read(a, i) represent the read operation and a{i<v ) or 
write(a, i, v) the write operation; the operator for equality is represented by =. 

The read and write operations are defined by 

• read: ARRAY x INDEX ->• ELEMENT and 

• write: ARRAY x INDEX x ELEMENT -> ARRAY. 

The axioms of the theory of arrays are 

1. the axioms of equality (symmetry, transitivity and refilexivity), 

2. \/a,i,j : i = j — >• read(a, i) = read(a,j) (array congruence), 

3. \/a,v,i,j : i = j read(write(a, i,v), j) = v (read- over- write 1) and 
4 ■ \/a,v,i,j : read(write(a, i, v), j) = read (a,j) (read- over- write 2). 

The theory of arrays consists of the equality operator = and two functions - read 
and write. The equality operator, in combination with the axioms of equality, state 
that a is equal to c if a is equal to b and if b is equal to c, for the arrays a, b and c. 

An array a and an index i are the inputs of the read operation; the output is a 
value v. It will be said that the value v will be read from an array a at an index i - 
denoted as v = read(a, i) or v = a[i). 
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On the contrary, an array a, an index i and a value v are the inputs of the write 
operation; the output will be a new array a' , which encodes logically the previous 
write operation. It will be said that the value v will be written into an array a at an 
index i - denoted as a' = write(a, i, v). The theory of arrays works in a functional 
manner (cf. Section 3.6 [52 ); for this reason, a new array will be returned that 
encodes the previous write access. 

For instance, let a be an array, i and j indices and, last, v and w values. Additionally, 
let i 7 ^ j and v j - w. Then, two write operations into a can be represented as 
a' = write(a, i, v ) and a " = write(a / , i, w ) = write(write(a, i, v),i, w). In other words, 
it holds that v == a"[i\ and w == a"[j] are true. 

These properties are modelled by the axioms. The axiom called array congruence 
defines that two read operations are equal if the same index will be read; i = j 
implies that a[i] == a[j] is true. The third axiom, read- over- write 1, states that 
if two indices are equal (i = j), then the value that will be read at an index j in 
the array a is identical to the preceding write of the value v into the array a at an 
index i. To illustrate, presuming the write operation a' = write(a,j,v), the formula 
v == a'[i] will be true if i = j. In contrast, if i =/= j, then v == a'[i\ will be false, 
but v == a'[j] be true, since the array indices are different. This characteristic is 
described by the fourth axiom, read-over-write 2. 


2.6 Bounded model checking 

Bounded model checking (BMC) is a technique used in the context of hardware and 
software verification. The approach of SMT-based bounded model checking is to 
prove a property p of a transition system M, which is unrolled up to a specified 
bound k. Then, an SMT solver will decide a formula consisting of the negation of the 
property p and the k times unrolled transition system M. If the formula is satisfiable, 
the property is proven to be wrong 1 13 . In the following, the application of BMC 
will be limited to software; the definitions and concepts will be based on the work by 
Sinz, Falke and Merz (38;|. 

The main idea of BMC is to create a transition system of a sequence of program 
statements. This transition system will be unrolled k times and translated into an 
SMT formula. Unwinding this system with a bound k connotes that loops in the 
sequence of program statements will be unrolled k times. As a result, the unrolled 
transition system of the sequence of program statements can be represented as a 
directed acyclic graph. Checking properties for an unbounded transition system of 
a program is undecidable because of its Turing completeness. On the other hand, 
checking properties for a bounded transition system is decidable, since such a system 
is a finite state machine; properties will be checked for finite program runs. 

For an SMT formula prog, which represents an unrolled transition system of a 
sequence of program statements, properties of the program statements will be applied 
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as preconditions and postconditions. Preconditions describe properties that will be 
assumed to be fulfilled; postconditions describe properties that must be fulfilled. 

Definition 2.6.1 (precondition, postcondition). Let prog be an SMT formula 
of an unrolled transition system of a sequence of program statements. An SMT 
formula precondition encodes an assumption that must hold for prog; an SMT 
formula postcondition encodes the property or assertion that must hold. 

Let prog be an SMT formula of a sequence of program statements. In addition, let 
preconditions be an SMT formula of a conjunction of assumptions that are assumed 
to hold for certain program statements. Last, let postconditions be a conjunction of 
properties that have to be proven. Then, the formula p = preconditions A prog A 
-i postconditions will be decided by an SMT solver. If p is unsat isfiable, then it is 
proven that the properties hold in the k times unrolled transition system, which is 
represented by prog as well as by the corresponding preconditions; in other words, 
the properties have been proven for a limited amount of runs of the sequence of 
program statements. If p is satisfiable, then the returned model is a counterexample, 
which proves that the asserted properties do not hold. 


2.7 Miasm’s intermediate representation 

Miasm [60 1 is an architecture- independent binary reverse engineering framework, 
which is licenced under GPLv2 1 61 and written in Python. It supports, amongst 
others, the architectures x86 [62 , x86__6f [62], ARMv 7 (63) , AArch64 (64] and 
MIPS32 [65]; for each architecture, it uses ‘its own disassembler [. . . ] and instruction 
semantic [s] ’ [60 . Additionally, Miasm facilitates the analysis of directed graphs, 
symbolic execution and code emulation, which is based on just-in-time compilation 
build upon TinyCC [66]. The symbolic execution operates on the level of basic 
blocks; in relation to the code emulation, functions of the application programming 
interface (API) can be emulated. In particular, Miasm uses its own intermediate 
language (IL) or intermediate representation ( IR ) called Miasm IR, which includes 
a translation into SMT formulas for the SMT solver Z3 [20 . In the following, the 
Miasm IR will be introduced. If not stated otherwise, the description of Miasm IR 
will be based on the work by Fabrice Desclaux [67] and the related source code [68] . 

Miasm IR is an architecture- independent intermediate representation and designed for 
binary program analysis and vulnerability research. It is explicit and based on mod- 
ular expressions. There exist eight categories of expressions that represent integers, 
variables, assignments, conditional statements, memory access, operations, extraction 
of bits and composition of expressions. Expressions from different categories may be 
combined. 

Definition 2.7.1 (Miasm IR). The intermediate representation of Miasm, Miasm 
IR, consists of modular expressions. An expression may be part of another expression; 
expressions may be nested. There exist eight categories of expressions. These are 
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1. Exprint, used to represent constants, 

2. Exprld, used to represent variables, 

3. ExprAff, used for assignments, 

4 ■ ExprCond, used for conditional statements, 

5. ExprMem, used for memory access, 

6. ExprOp, used for applying operators on expressions, 

7. ExprSlice, used to extract bits from an expression and 

8. ExprCompose, used to compose or concat expressions. 

The size of an expression is included in its definition. If two expressions, e\ and ei, 
will be combined, then the combined expression is conforming to the sizes of e \ and 
62- 

As a result of the compact but modular design, constants, variables and memory 
expressions can be distinguished. Only assignments modify other expressions. If a 
memory expression is on the left-hand side of an assignment, it is a memory write; 
if it is on the right-hand side of an assignment, it is a memory read. Conditional 
statements are of the form condl a : b — if cond is true, then a will be returned, 
otherwise b. In operations, operands must have the same size; one operator and at 
least one operand will be expected. Operations can be unary operations, such as ~<a 
and —a, or binary operations, such as a + b and a ■ b, whereby a and b are expressions. 
It is also possible to define operations for a variable number of arguments. Then, it 
will be represented as a function, for instance gcd(ai, . . . , a n ), whereby ai, . . . , a n are 
expressions and gcd() is the operator. Last, expressions may be sliced and composed. 
For instance, a register EAX (represented as variable) can be expressed as the slice 
of the first 32 bits of the register RAX, which has a size of 64 bit. Contrary to that, 
two variables with a size of 32 bit can be composed to a variable with a size of 64 bit. 

While assembly code is implicit, the intermediate representation is explicit - every 
assignment has exactly one effect. Each assembly instruction will be transformed 
into a list of expressions. For instance, POP RBP will be transformed into [RSP = 
(RSP+0x8) , RBP = @64 [RSP] ] ; RSP will be incremented by 0x8 and RBP will be 
assigned to the memory value at the address that is stored in RSP. In Miasm IR, 
the list of expressions that are related to one assembly instruction will be evaluated 
simultaneously; the intermediate representation of an assembly instruction describes 
the input and output state of that instruction. The right-hand side of the assignments 
in the intermediate representation describes the input state, the left-hand side the 
output state. To put it differently, the order of the intermediate expressions is 
insignificant as long as they belong to the assembly instruction that is evaluated 
currently |69|. 
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Lastly, a few characteristics of Miasm IR will be mentioned. Miasm IR’s instruction 
pointer, IRDst, is always the last instruction of a basic block in the intermediate 
representation. A basic block in the assembly code may be translated into several 
basic blocks in the intermediate representation. Each function call is represented as 
a separate basic block in the intermediate representation |69 
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3 Bounded model checking for input crafting 


Building on the preceding established theoretic foundations, this chapter describes 
the process as the input crafting problem can be addressed with bounded model 
checking (BMC). First, the general idea of using bounded model checking for input 
crafting will be introduced. After that, its individual steps will be outlined. The 
prerequisites, function discovery and API call handling, as well as the process of 
unrolling the transition system and rewriting the intermediate representation, will 
be discussed. Then, the most important step will be exposed in the context of SSA - 
encoding the control flow into a logical formula. Last, the memory model will be 
explained and the building of an SMT formula will be illustrated. 

In contrast to the previous chapter, definitions in this chapter will be informal; 
they will be used to summarise concepts, instead of providing formal frameworks. 
The same holds for algorithms, which will be mentioned where necessary, but not 
discussed in detail. 


3.1 Introduction 

In this section, bounded model checking for binary input crafting will be introduced. 
First, the general idea will be derived. Second, the process of binary BMC will be 
exposed. Thereafter, a concrete framework that executes the described process will 
be presented. Last, the underlying assumptions for this task will be discussed. 


3.1.1 General idea of BMC-based input crafting 


As described in Section |2.3[ symbolic execution is a common technique for input 
crafting. To locate a user input which directs the control flow to a specific basic 
block in the control flow graph, constraints will be derived by evaluating symbolically 
possible execution paths. If the constraints for a path can be satisfied, then the 
satisfying model represents an input with the required characteristic. One main 
limitation is the exponential growth of execution paths in the number of branches, 
known as the path explosion. On these grounds, the path selection problem leads to 
the question of which path should be evaluated first (cf. Definition 2.3.3). 


To address path explosion and alleviate path selection, in relation to input crafting, 
the number of paths can be reduced by static data flow analysis. Only paths that are 
known to be influenced by user input have to be evaluated. In the worst case, static 
data flow analysis has to be performed on the entire control flow graph to acquire 
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this knowledge. In this case, the user input is defined in the first basic block of a 
function. 


As stated in Section 2.5.3} SMT solvers are strong in deductive reasoning, especially 
when applied to real-world problems. On that account, it is feasible that SMT solvers 
are strong in performing data flow analysis; the question that arises is whether or 
not static data flow analysis can be encoded as a logical problem that will be decided 
by an SMT solver. Extending this question, it can be asked if it is possible to find 
a solution to the path selection problem with an SMT solver. Furthermore, the 
generalised question asks if it is operable to transform the entire control flow graph 
into a formula of first-order logic and decide the input crafting problem by solely 
relying on an SMT solver, which tracks the dependencies of the user input, chooses 
a path and returns the chosen path and a satisfying user input, if the problem is 
satisfiable. 


This question can be formulated as an instance of bounded model checking. With 
respect to Section 2.6 let the control flow graph be the transition system of a 


sequence of program statements. Then, the input crafting problem can be described 
as the property that a defined basic block must be visited. To put it differently, 
an SMT solver that decides a formula consisting of the conjunction of the unrolled 
control flow graph and the mentioned property, which is encoded as postcondition, 
decides the input crafting problem. 


3.1.2 Process of BMC-based input crafting 

The idea to perform bounded model checking on the binary level is inspired by 
work by Sinz, Falke and Merz 1 38 1 , who conduct bounded model checking based on 
the intermediate representation of LLVM (Low Level Virtual Machine) [18 for the 
purpose of discovering memory errors in programs written in C. Subsequently, the 
process of bounded model checking for binary input crafting will be outlined; the 
essential steps will be described explicitly in the succeeding sections. 

Definition 3.1.1 (bounded model checking for binary input crafting). Let 

a sequence of program statements be assembly code. Additionally, let all calls to 
functions, the assembly code and the control flow graph of those functions be known. 
Lastly, let both, the basic block b\ , which defines user input, and 62 , the basic block 
to which the control flow must be directed by user input, be known. Then, bounded 
model checking for binary input crafting consists of the following steps. 

1. transformation from assembly into an intermediate representation 

2 . directed acyclic graph generation 

3. SSA transformation 

4- translation into an SMT formula prog 
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5 . defining preconditions and postconditions 

6 . SMT solving of ip = preconditions A prog A postconditions 


The preconditions are optional and define assumptions that have to hold. The 
postconditions describe the property that 62 must be visited. A model for <p describes 
an execution path p from b\ to 62 and all constraints that define the control flow on 
the path p. 


As described in the previous section, bounded model checking operates on a control 
flow graph, in the context of this work. Therefore, the program statements - the 
assembly instructions - have to be derived by disassembling the relevant parts of 
the binary program. Further, a function discovery has to be performed. Since the 
control flow graph will be unrolled, function starts, ends and calls have to be known. 
The same holds for calls to API functions because those functions could influence 
the control flow and have to be considered and handled. For reasons described in 


Section 2.7 the derived assembly instructions will be translated into an intermediate 
representation. On this basis, the control flow graphs can be constructed for the 
intermediate representation. Therefore, it will be assumed that a control flow graph 
exists for each discovered function. In addition, it will be assumed that it is possible 
to associate a basic block in the intermediate representation with the corresponding 
basic block in the assembly code and vice versa. 


Hence, after the control flow graphs of the functions are derived, the transition system 
will be unrolled k times. For that, loops will be unwinded k times and functions 
will be inlined. As a result, the unrolled transition system will be represented by 
a directed acyclic graph. Subsequently, the intermediate representation has to be 
rewritten, such that the program semantics will be preserved. The rewritten sequence 
of program statements represents all possible execution paths for the unrolled control 
flow graph. In other words, this sequence represents all possible paths of a finite 
program run. 


In the next step, the directed acyclic graph will be prepared so that it can be 
translated into an SMT formula. For that, SSA will be applied to the graph. As a 
result, each assignment will be unique. Lastly, the control flow must be encoded into 
a logical formula. At this stage, the directed acyclic graph will be represented by a 
set of constraints in the intermediate representation. These constraints, as well as 
the encoding of the control flow, will be translated into SMT formulas and combined 
to one large formula in conjunctive normal form; the directed acyclic graph will be 
represented by an SMT formula prog. 


Then, the input crafting problem will be encoded as the condition that a certain basic 
block must be visited. This will be handled by a variable visit, which is true when 
the block has been visited; otherwise, it is false. In relation to Section 2.6, an SMT 
formula is of the form = preconditions A prog A -1 postconditions . Therefore, the 
input crafting problem has to be defined as postconditions = ( visit == 0) in order 
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to receive a satisfying model for the input crafting problem. For the sake of simplicity, 
it will instead be encoded as p = preconditions A prog A postconditions , whereby 
the postcondition will be defined as postconditions = ( visit == 1). Additionally, 
the variable preconditions may be set to model assumptions; to exemplify, it may 
be set to model restrictions on the user input. 

Finally, the SMT solver decides p. If it returns UNSAT, then there exists satisfying 
input for the chosen bound; the bound can be increased and the procedure repeated. 
Otherwise, a satisfying user input will be returned. 


3.1.3 Framework 


Cylyx is a cross-platform framework for bounded model checking on the binary level 
that has been designed and realised as essential part of this work. It is written in 
Python and implements the concepts described in the following sections. Before 
these concepts will be elucidated, some characteristics of the implementation will be 
stressed. 


The emphasis of Cylyx is to realise the core concepts of bounded model checking. 
Thus, in relation to the previous section, the focuses are unwinding the transition 
system, generating the SMT formula and defining postconditions. In contrast, 
prerequisites, such as function discovery, are implemented fragmentarily since these 
tasks can be performed by more powerful tools as IDA [71 . Those tools are not 
utilised, by default, in order to minimise required dependencies. Therefore, Cylyx 
supports these prerequisites, but also allows the integration of other frameworks. 


Based on Miasm, Cylyx uses Miasm’s disassembler, graph analysis and, especially, 
Miasm’s intermediate representation, Miasm IR, as well as its translation into SMT 
formulas (cf. Section 2.7). Building on this, Cylyx performs graph transformations, 
such as function inlining and loop unrolling, rewrites the instructions to represent 
the program semantics of the directed acyclic graph, transforms this graph into SSA 
and translates it into an SMT formula. Cylyx is architecture-independent due to its 
implementation of the core concepts of bounded model checking, as well as Miasm’s 
ability to translate assembly code of different architectures into Miasm IR. In the 
context of this work, parts of Cylyx have been submitted to and are now included in 
Miasm. All contributions are listed in Appendix [X] The submissions include graph 
algorithms, bug fixes and extensions of the translation into SMT formulas. 


3.1.4 Assumptions 

To craft inputs for assembly code, some prior knowledge of the assembly code will 
be assumed. This knowledge includes the basic block in which user input will be 
defined as well as the basic block to which the defined user input must direct the 
control flow. For instance, if a function / has three input parameters a, b and c, if 
this function will return 1 or 0 and if the input for a and b should be crafted such 
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that / will return 1, then knowledge of the assembly code and control flow graph of 
/ will be presumed. In addition, it will be inferred to be known which basic block 
will return 1 and that a and b are defined by user input. If the parameter c also 
influences the control flow, it must be known as c is defined outside of /. This will 
be encoded as a precondition. 


3.2 Prerequisites 

This section describes the prerequisites needed to perform bounded model checking 
for binary programs and methods to cope with those prerequisites. First, the subject 
function discovery will be addressed; second, the detection and handling of API 
functions will be explained. 


3.2.1 Function discovery 


As described in Section 3.1.2 bounded model checking on the binary level requires a 
precise description of the program semantics; it requires every assembly instruction 
of the relevant parts in the binary. For this, in the case of function calls, functions 
that will be called have to be inlined (cf. Section 3.3.1). This necessitates knowledge 
of function calls, as well as knowledge of the control flow graphs and the assembly 
code of called functions. The process of detecting functions will be called function 
discovery. 


Definition 3.2.1 (function discovery). Let a sequence of program statements be 
disassembled assembly code. Then, function discovery describes the task of grouping 
assembly instructions such that they represent functions of high-level programming 
languages. 


It is well-known that distinguishing between code and data is undecidable. Therefore, 
in general, disassembling is undecidable (cf. Section 1.2.1 [72]). This implies that func- 
tion discovery itself is undecidable, since it relies on disassembled code. In practice, 
functions can be detected with heuristics. For instance, IDA and BYTEWEIGHT 
are known to implement powerful heuristics for function discovery. 
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Function discovery performed by Cylyx relies on two assumptions; a function call 
jumps to the beginning of a function and there exists a return statement at the end of 
a function. The disassembled code between the beginning and the end of the function 
describe the semantics of the function. This basic approach works for unoptimised 
code without indirect function calls. For more powerful function detecting, IDA or 
radare2 [74 can be used; the results have to be prepared such that they can be used 
by Cylyx. In other words, Cylyx’s function discovery can be replaced by a more 
powerful one, if some manual work will be done. 
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3.2.2 API functions 


Concerning the previous section, to describe precisely the program semantics, API 
functions must also be considered. Since API functions are often located outside of 
the binary and depend on the architecture, they cannot be included straightforwardly 
as functions within the same binary. Therefore, they have to be handled somehow. 
In the following sections, the detection of API calls will be described. Then, an 
interface for handling API functions will be characterised. 

An API call is a function call that calls an API function. Often, API calls are calls 
to thunk functions, which jump into API functions. Thunk functions are functions 
that ‘contain nothing but a single jump statement’ (Chap. 20, p. 429 |75]). 

Cylyx performs API call detection as follows. Firstly, it parses the imports of the 
binary. Then, it locates the thunk functions. (In the case of ELF (Executable and 
Linking Format) [76 files, it parses the binary. For PE (Portable Executable) [77] files, 
it relies on an IDA script which finds thunk functions. Afterwards, these functions 
have to be defined manually for Cylyx.) Lastly, a call to a thunk function which 
jumps into an API function will be detected as an API call. 

As described above, Cylyx provides an interface for API call/function handling. 
For each API function, the handling has to be defined manually. Cylyx supports 
removing API calls from the control flow graph, inlining API functions and modelling 
API functions in an SMT formula. These will be discussed in the next sections; the 
third approach is not implemented, but considered as future work. 


API functions that do not affect the control flow can be removed from the control 
flow graph. For instance, the functions printf or puts do not modify the control 
flow as long as their return value will not be used after the API call. In such a case, 
the program semantics do not depend on those functions. Since Miasm IR represents 
a function call as a separate basic block (cf. Section 2.7), this call block can be 
removed from the control flow graph. 


The second approach, the inlining of API functions, is based on the idea that, if 
an API function can be represented as a single control flow graph without calls 
to other functions, then it can be treated as a function within the binary and be 
inlined (cf. Section 3.3.1). This method requires manual preparation, which has 
to be done for every API function of interest and for each architecture. Firstly, an 
API function and all functions that will be called within this function need to be 
disassembled. Secondly, these functions must be inlined such that the API function 
will be represented as one large control flow graph. Disassembling API functions 
connotes operating on optimised code, which may be multithreaded or may contain 
indirect function calls. Therefore, it cannot be applied to a large scale. However, if 
this has been done for a set of API functions, then those API functions can be inlined 
without further preparations. Cylyx includes scripts for facilitating this process, 
such as an IDA script for function discovery or a script to prepare API functions for 
inlining. 
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The last option for API call handling is not implemented. However, this approach 
is based on the idea to model the behaviour of an API function as an SMT for- 
mula, instead of building it on the basis of assembly code. This might enable an 
architecture-independent handling and may be more efficient than the second ap- 
proach. For instance, Cylyx models low-level memory management in SMT formulas 
(cf. Section 3.6); Sinz, Falke and Merz [38] describe an approach for modelling malloc 

demonstrate an abstract framework 


and free. Lastly, Vinod Ganapathy et al. [78] 
for modelling API functions and apply it to printf . 


3.3 Graph transformations 

Until now, assembly code has been derived, function discovery has been performed 
and the handling of API functions has been defined. Thus, for every function, there 
exists a control flow graph that has been constructed from assembly code. In the 
next step, the assembly code will be translated into the intermediate representation 
of Miasm, Miasm IR. Henceforth, operations will be performed either on control flow 
graphs which have been constructed from the intermediate representation or on the 
intermediate representation itself. 

This section describes the process of unwinding the transition system. Since a control 
flow graph represents the transition system, unrolling will be achieved by graph 
transformations. In the following, methods for function inlining, unrolling reducible 
loops and unrolling irreducible loops will be demonstrated. Then, a process which 
combines these methods will be described. As a result, the transition system will be 
a directed acyclic graph. 


3.3.1 Function inlining 


Function inlining describes the process of replacing function calls with the body of 
the called function. Since a function call will be represented as a separate basic 
block in Miasm IR (cf. Section 2.7), function inlining can be defined, informally, as 
replacing a function call block in a control flow graph with the control flow graph of 
the called function. 


Definition 3.3.1 (function inlining, caller, callee). Let G = (V, E) be a control 
flow graph of a function f\ and let a basic block b e V represent a function call to a 
function f 2 . Additionally, let the control flow graph of f 2 be G' . If b will be replaced 
by the control flow graph of fi in G, then it will be said that fi will be inlined into 
f\ . E will be extended with edges from the direct predecessors of b to the heads of G' 
and with edges from the leaves of G' to the direct successors of b. f\ will be referred 
to as the caller and /2 will be referred to as the callee. 


This will be illustrated in Figure 3.1 In the control flow graph G = (V,E) with 


V = { a , b, c} and E = {(a, b), (b, c)} of a function / 1 , the basic block b calls a function 
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Figure 3.1: Replacing a function call with the callee’s control flow graph 


f 2 ; fi is the caller and f 2 the callee. The control flow graph of f 2 is G' = (V',E') 
with V 1 = {b 0 ,bi,b 2 ,b 3 } and E' = {(b Q , bi), {b 0 , b 2 ), (&i, 63), (b 2 , 63)}. b 0 is a head 
of G’ and 63 a leaf. To inline f 2 into f\, the direct predecessors of b have to be 
connected with the heads of f 2 and the leaves of f 2 have to be connected with the 
direct successors of b. Therefore, the edges ei = (a, bo) and e 2 = (63, c) will be added 
to E. 


3.3.2 Unrolling reducible loops 

Unrolling a loop up to a certain bound k characterises a graph transformation which 
eliminates a loop in a control flow graph without modifying the program semantics 
for k loop iterations. To put it differently, loop unrolling removes a loop in a control 
flow graph; however, the modified control flow graph represents k iterations of the 
loop that has been removed. 

Definition 3.3.2 (loop unrolling). Let G be a control flow graph for a sequence of 
program statements. It will be said that a loop l is unrolled k times, ifG is transformed 
such that l no longer exists in G, but the program semantics are preserved for k 
iterations of l . 
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As stated in Section |2.1.5[ a reducible or natural loop is a loop with a single loop 
header. The loop header dominates every node in the loop. Therefore, the existence 
of a back edge - an edge from a node to a dominator of this node - implies the 
existence of a reducible loop. To unroll a reducible loop, the back edge has to be 
removed. Then, the nodes of the loop body will be duplicated k times for k loop 
iterations. In every case, the edges need to be set properly. 



Figure 3.2: Unrolling a reducible loop k = 2 times 


For instance, in Figure 3.2 the node a dominates 6 , the edge eb = ( 6 , a) is a back 
edge and l = {a, b} defines the loop body of the reducible loop l. Since the loop is 
unrolled k = 2 times, the nodes a o and bo represent the first loop iteration; ai and b\ 
represent the second loop iteration. To preserve the program semantics, the loop 
edges are set properly for every iteration. The edge (a, c ) is represented by (ao, c) for 
the first and (a i, c) for the second loop iteration. The same holds for ( b , c). (b, a), the 
back edge, has been removed; instead, the next loop iteration is defined by (bo,ai). 
Since the walk w = (ao, bo, ai,b\) already represents two loop iterations, there is no 
additional edge ( 61 , 02 ) for a third loop iteration. In other words, the loop condition 
no longer holds. This has to be considered when the intermediate representation is 


rewritten, as will be described in Section 3.4 
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3.3.3 Unrolling irreducible loops 

Contrary to reducible loops, irreducible loops are loops with multiple loop headers. 
Therefore, the dominance relations (cf. Section 2.1.4) for irreducible loops are 


different than those for reducible loops. As a result, the approach described in the 
previous section cannot be utilised for irreducible loops without difficulty. 

a method known as node splitting enables the 


As mentioned in Section 2.1.5 


transformation of irreducible loops into reducible loops in an exponential running 
time. Since the irreducible loops have to be unrolled, this approach is not required. 
Instead, for each loop header h, it will be assumed that h is the only loop header. 
Then, the dominance relations hold for the loop dominated by h: this loop can be 
unrolled with the method described in the previous section. Again, in relation to the 
program semantics, edges need to be set properly. 




Figure 3.3: Unrolling an irreducible loop k = 2 times 

To illustrate this concept, Figure |3.3| shows an irreducible loop between b and c; 
it holds neither b dome nor cdomf>. Therefore, b and c are both loop headers of 
l = {b, c}. To unroll, first, assume that b is the loop header. Then, (c, b) is a back 
edge and b dominates c. As a result, l is a reducible loop that can be unrolled, 
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as described in the previous section. The edges (&o,d) and (62, d) represent the 
edge (6, d); (60, ci) and (62,03) represent the edges within the same loop iteration 
and (01,62) represents the back edge. Again, since (60,01,62,03) represents two loop 
iterations starting in 6 and since c has no other outgoing edges, the semantics for 
the loop starting in 6 are preserved. Next, assume that c is the loop header. Thus, 
(6, c) is the back edge and c dominates 6. Thereby, again, l is a reducible loop and 
can be unrolled in the same way. 


3.3.4 Procedure 


In this section, a procedure which combines the previously described graph trans- 
formations to unroll a transition system will be outlined; as a result, the unrolled 
control flow graph will be a directed acyclic graph (cf. Definition 2 . 1 . 6 ). After that, 
Cylyx’s approach will be presented. The graph algorithms used by Cylyx will only 
be mentioned, since they can be replaced by others with a similar running time and 
since they are out of the scope of this work. 


First, to unroll a control flow graph with function calls k times, function inlining will 
be performed. Consequently, the resulting control flow graph does not contain any 
function calls; reducible and irreducible loops may exist. Therefore, irreducible loops 
will be unrolled k times. Since the dominance relations are not explicit in irreducible 
loops, they will be unrolled first. After that, each loop in the transformed control 
flow graph is reducible. Thus, reducible loops will be unrolled k times. Subsequently, 
the transformed graph is a directed acyclic graph without functions calls. 


Cylyx extends the described approach for API functions. As stated in Section [ 3 . 2.2 
API functions can be inlined or removed. First, defined API functions will be removed 
from the control flow graph. For this, the basic block which calls the API function 
(to be precise, the basic block that calls the thunk function, which jumps into the 
API function) will be removed; its direct predecessors will be connected with its 
direct successors. Second, defined API functions will be inlined. As of now, the 
control flow graph does not contain any API calls. Third, function inlining will be 
performed. As a result, the transformed control flow graph may only contain loops. 


Loop detection will be performed by locating strongly connected components (based 
on an iterative version of Gabow’s path- based algorithm [43 ) with at least two 
distinct nodes. In this way, in relation to Section 2 . 1.5 outermost loops will be 
detected. Although outermost loops do not consider loop nesting, the program 
semantics will be preserved; if the unrolling bound k does not cover the required 
number of iterations of the innermost loop, k can be increased. 


To localise irreducible loops, all back edges in the control flow graph will be detected 
and removed. (Cylyx includes algorithms used to locate back edges and natural 
loops, described by Alfred V. Aho et al. (cf. Section 9 . 6.6 [4]).) Since a back edge 
defines a natural loop, removing a back edge is equivalent to unrolling a reducible 
loop by k = 1 . In other words, reducible loops will be eliminated in the control flow 
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graph. Therefore, the loop detection described above will only detect irreducible 
loops, which will be unrolled k times. Then, on the transformed control flow graph, 
the back edges will be re-added. Ultimately, the only possible loops are reducible 
loops, which will be detected and unrolled k times. 


3.4 Rewriting the intermediate representation 


Before the directed acyclic graph can be transformed into SSA, the unrolling of 
the transition system has to be finalised. So far, the intermediate representation 
and the control flow graphs of each function, as well as the directed acyclic graph 
of the unrolled transition system, are known. In the next step, the intermediate 
representation must be rewritten such that the program semantics will be represented 
by the transformed control flow graph. Additionally, the intermediate representation 
will be prepared for SSA transformation. 

In order to rewrite the control flow, Cylyx performs a depth-first traversal on the 
directed acyclic graph, starting at the head. For each node, the corresponding basic 
block b will be analysed. If the direct successors of the current node in the directed 
acyclic graph differ from the direct successors of the basic block b, the last statement 
of the basic block, which directs the control flow, will be rewritten. Additionally, 
logical implications will be derived, which are dependent upon the jump condition. 
These implications will be used to encode the control flow graph, as will be described 
in Section | 3 . 5 . 2 | If a basic block corresponds to multiple nodes in the directed acyclic 
graph - for instance, if this basic block is part of an unrolled loop - then there will 
exist multiple rewritten copies of the basic block’s intermediate representation, one 
for each corresponding node. 

To exemplify, b is the current basic block and 65 is the corresponding node in the 
directed acyclic graph. Additionally, let the last statement of b with the jump 
condition j be j ? c : d, whereby c and d represent the direct successors of b in the 
control flow graph. Let b and c be part of the same loop, but not d. Then, the 
intermediate representation of b will be copied to create a new basic block 65. The 
jump statement of 65 will be rewritten to j ? C5 : d, since the direct successors of 65 
are C5 and d. Additionally, logical implications will be derived. Assume that the 
instruction pointer IP will be set to depend on the jump statement, for instance 
IP = j ? C5 : d. Then, the implications IP == C5 =>■ j == 1 and IP == d =>■ j == 0 
will be derived; if C5 is the next basic block that will be visited, then the jump 
condition must be true, otherwise, this condition must be false. 


Similarly, let b be the same basic block and bg be the corresponding node in the 
directed acyclic graph. Let bs be the last loop iteration and (6, c) the back edge of this 
loop. Then, in relation to Section 3 . 3.2 bs will only have one direct successor, d, in the 
directed acyclic graph; however, c and d are the direct successors of b. For the jump 
statement of b, IP = j ? c : d, the implications IP == d and IP == d => j == 0 
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will be derived; since the back edge cannot be taken, the next block must be d and 
the jump condition has to be false. 


Last, certain instructions will be rewritten in order to prepare for SSA transformation. 
For instance, as will be exposed in detail in Section 3.6 memory instructions will 
be rewritten such that memory access can be modelled as a load/store architecture. 
Then, the variable M represents global memory access and will be transformed into 
SSA. 


3.5 Static single assignment 

In this section, SSA transformation on a directed acyclic graph will be depicted. As 
a consequence of SSA, each variable definition is unique; the control flow graph will 
be transformed into a set of constraints. For this, one additional step has to be 
performed - the control flow has to be encoded. In the following, the construction of 
SSA and the encoding of the control flow will be outlined. Since constructing SSA 
is an intricate process, which is mostly based on algorithms described in standard 
literature, only the main ideas of this process will be mentioned [41 . Aside from 
that, the focus of this section rests upon encoding the control flow logically. 


3.5.1 Construction 


Constructing SSA relies strongly on graph-theoretic concepts, which are described 
in Section |2.1[ As stated in Section |2.1.6[ the two main tasks of SSA are renaming 
variables and inserting T-functions. Since variable renaming will also rename variables 
in <h-functions, these functions must first be inserted. If not stated otherwise, the 
following steps are based on the work by Cytron et al. |4l]. 


In relation to Definition 2.1.25 a <h- function distinguishes the variable assignments 
of different execution paths in converging paths. Therefore, it is possible to insert 
a <h-function for every variable in a basic block in which path convergence exists. 
However, Cylyx builds on an SSA construction, which is known as minimal SSA. 
In minimal SSA, the amount of <L-functions is reduced since the insertion relies on 
dominance relations, especially on the dominance frontier (cf. Section 2.1.4). As a 
consequence of Definition 2.1.14 the dominance frontier of a basic block b defines 


the set of nearest basic blocks that are not dominated by b; a variable v, which 
is defined in b, may be redefined in the execution paths that do not depend on 
b. Therefore, a 4>-function for a variable v will be inserted into each basic block 
which is in the dominance frontier of a basic block b, for each defined variable v in b. 
Cylyx determines the dominance frontier of a node based on an algorithm by Cooper, 
Harvey and Kennedy ||44|. 


In the next step, on the directed acyclic graph, the variables will be defined and 
the ^-functions for the variables will be updated with the latest variable definitions. 
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In general, without delving into much detail, the worklist algorithm for variable 
renaming performs a depth- first traversal on the dominator tree (cf. Definition 2.1.13 ), 
starting at the head of the graph, and carries out the following steps. First, the 
variable definitions will be renamed for all ^-functions in the current basic block. To 
exemplify, the variable definition v = </>(...) will be transformed into v\ = (f >(. . . ). 
Second, for each assignment, all variable uses on the right-hand side will be renamed 
for non-^-functions. Third, all variable definitions on the left-hand side will be 
renamed for each assignment. For instance, the assignment a = a + b will first 
be renamed to a = ao + & 2 i then to a\ = ao + 62 - Last, 4>-functions in the direct 
successors of the current basic block in the control flow graph will be updated on the 
right-hand side. For example, assume that u 3 is the last definition of a variable v. 
Then, if there exists a ‘L-function for v in the direct successor of the current basic 
block, this <h-function will be updated with U 3 ; v = 4 >(vi,V 2 ) will be updated to 
v = cj)(v i,u 2 ,u 3 ). 

Finally, some technical characteristics will be mentioned. Initially, variables in an 
assignment will first be updated on the right-hand side and then on the left-hand 
side, since SSA defines a def-use chain (cf. Definition 2.1.24) that starts with a 
variable definition of a variable v and ends with a reassignment of v. If an assignment 
v = v + 1 for a variable v will be transformed into SSA form, the right-hand side 
must be transformed first, since the definition of v on the left-hand side starts a new 
def-use chain for v. Second, Cylyx computes the dominator tree of a control flow 
graph by defining tree edges as edges from an immediate dominator of a node v to v. 
Therefore, Cylyx includes an algorithm which computes the immediate dominators 
of a directed graph. Third, the logical implications, which have been derived in the 
context of rewriting the intermediate representation (cf. Section 3.4), will also be 
transformed into SSA. Last, the final stage in the process of variable renaming ~ 
updating 4>-functions - will be extended and utilised to encode the control flow of the 
directed acyclic graph into logical formulas. This will be described in the following 
section. 


3.5.2 Control flow encoding 


This section describes how SSA can be used to encode the control flow logically such 
that it can be transformed into an SMT formula. First, the concept of ^-functions 
will be extended. Then, it will be presented as these extensions will be generated. 
The following ideas are based on the work by Sinz, Falke and Merz [38 1 . 


As stated in Section 2.1.6 a T-function for a variable v returns the variable for the 


current execution path. In general, 4>-functions are not defined in a concrete manner. 
However, bounded model checking requires a concrete modelling of the program 
semantics. This will be achieved by extending the concept of 4>-functions with edge 
conditions that represent the current execution path. 
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Definition 3.5.1 (4>- functions for bounded model checking, edge condi- 
tions). Let G = (V, E) be a directed acyclic graph. For each edge e* G E there 
exists a condition Ci that can be true or false. If c t is true, then e* is in the current 
execution path. Otherwise, Ci is false. For a variable v, a -function for bounded 
model checking is of the form 

Vj := $((ci, vi), (c 2 ,v 2 ), ( c n ,v n )) 

and can be translated into a statement of nested conditions, which has the form 
Vj := ci?vi : [c 2 ?v 2 : (. . . (c n _ i?u n -i : v n ) ...)) 


with 1 < n < j. 


To illustrate, in Figure | 3 . 4 [ let the variables v\, v 2 and V3 be the last variable 
definitions of v in the corresponding execution paths labelled by the edge conditions 
ci, c 2 and C3. Then, the ^-function tq = 4 >(vi,v 2 , V3) for v in the basic block a will be 
extended to V4 = (j>((ci, v\), (c 2 , v 2 ), (03, U3)). This can be translated into the logical 
formula C4 = ci?ui : (c 2 7 v 2 : V3). The edge conditions are mutually exclusive, since 
there exists only one execution path at a specific point in time. 



Figure 3.4: Control flow encoding based on edge conditions 

As a consequence, a visit flag for a basic block can be defined. A basic block is 
visited, if one of its incoming edge conditions is true. 

Definition 3.5.2 (visit flag). Let G = (V, E) be a control flow graph. Let b e V 
be a basic block and e\, e 2 , . . . , e n G E its incoming edges. Additionally, let c \, ... ,c n 
be the corresponding edge conditions. A basic block b is visited if its visit flag 

n 

visit_b = \J Ck 
k = 1 


is true. 
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To complete control flow encoding, the edge conditions must be set. Let 6 be a basic 
block with the visit flag visit_b, the jump condition j and the two direct successors 
si and S 2 - Next, assume that the instruction pointer IP determines whether the 
control flow will be directed to si or to S 2 with IP = j ? s i : S 2 - Then, the edge 
conditions will be defined as c Si = visit_bA(IP == s*) for e = (b, s*), with i e { 1 , 2 }. 
If b has only one direct successor s, then a jump condition does not exist. The 
instruction pointer is defined as IP = s. In this case, the edge condition will be 
defined as c e = visit_b A (IP == s) with e = (b, s). 

For instance, in Figure [3~i] let the instruction pointer IP for the jump condition j 
of a be IP = j? b : c. Then, C 4 is defined as C 4 = visit_a A (IP == b). In other 
words, C 4 is true if the next basic block is b and a is visited. Equally, C 5 is true if a is 
visited and the next basic block is c. 


Finally, this logical encoding of the control flow works as follows. The visit flags 
define a path from a basic block to the root of the directed acyclic graph. For 
instance, if the visit flag of b is true, C 4 has to be true since it is the only incoming 
edge of b. Therefore, a has to be visited. This implies that ci, C 2 or C 3 must be 
true. On the other hand, the derived block conditions (cf. Section 3.4) encode the 
control flow from the head of the directed graph down to the leaves. For instance, 
the derived block conditions for a are IP == b =>■ j == 1 and /P==c=^j== 0 . 
Therefore, the control flow is encoded for both directions. 


3.6 Memory model 


Memory management describes the strategy as memory access will be modelled. Based 
on Miasm IR, Cylyx rewrites the intermediate representation (cf. Section 3.4) such 
that it constitutes a load/store architecture. A load/store architecture characterises 
the property that exactly two operations exist to access memory 1 18 1 . 


Definition 3.6.1 (load/store architecture). In a load/store architecture, memory 
access will be performed by two operations, load and store. Load reads a value from 
memory; store writes a value into memory. 


Memory will be modelled architecture-independently as a global array, which is 
represented by a variable M ; Cylyx utilises a global memory model based on the theory 
of arrays (cf. Section 2.5.5). Memory access is represented by v = mem_read(M, 
address, size) for the read operation and by M = mem_write(M, address, v, 
size) for the write operation in the rewritten intermediate representation. Similar 
to the theory of arrays, a value v will be read from or written into an array M at 
an index address ; equally, a value v or a memory array M, which encodes the write 
access, will be returned. The only difference is the additional parameter size; a value 
that will be read from or written into global memory has a defined byte size. 


Cylyx implements a low-level memory model in SMT formulas. A memory array is a 
map of bit vectors with a size of l\ to bit vectors with a size of I 2 . Memory addresses 
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with a length of l\ will be mapped to bit vectors with a length of I 2 = 8; the memory 
model is byte-addressable (cf. Section 4.6 [79 ). The word size - the number of bits 
a processor architecture can process simultaneously (cf. Section 2.1 [79]) - defines 
the size of memory addresses. Then, memory reads and writes will be performed 
iteratively, byte per byte, until a value v of size s has been read or written. If the size 
s of a value v is not a multiple of 8 - if the size is not aligned (cf. Section 4.6 [79] ) 
- then v will be aligned. Each operation can be fulfilled endian-independently, on 
little-endian architectures as well as on big-endian architectures. 

For instance, on the x86 architecture, the word size is l = 32. Therefore, memory 
addresses with a size of 32 bit will be mapped to values with a size of 8 bit. Let v be 
a value with a size of l = 15 bit. Additionally, let addr be the memory address that 
contains v. Then, the expression v = mem_read(M, addr, 15) will be translated 
into v == [concat (read (M, addr + 1), read(M, addr))]}^. Since v has a size of l = 15, 
the size will be aligned to l' = 16. Then, two bytes will be read and concatenated in 
little-endian representation. Lastly, only the first 15 bits will be returned, which is 
denoted by [a]g 4 for a value a. 

The characteristics of this memory model facilitate an architecture-independent 
modelling of memory access. Additionally, since the memory array is a variable and 
since a memory write returns a memory variable, global memory can be transformed 
into SSA form. To put it differently, there exists a unique memory state for each 
program state. Lastly, the combination of SSA form and this memory model implies 
that an SMT solver handles memory aliasing. For instance, an SMT solver detects 
if [EAX] and [EBX + 0x8] point to the same memory location, since it knows the 
relations of EAX and EBX. 


3.7 Formula generation and SMT solving 


In the final step, the SMT formula consisting of the unrolled transition system, 
preconditions and postconditions will be created. Then, this formula will be solved. 
If it can be satisfied, there exists a solution for the input crafting problem for the 
defined bound k; otherwise, it is UNSAT. 


Each program statement in the intermediate representation in SSA form will be 
translated into an SMT formula based on Miasm’s translation for Miasm IR. Memory 
depends on the theory of arrays; variables are modelled in the theory of bit vectors. 
Therefore, the SMT formula will be a formula for the combined theory of quantifier- 
free bit vectors and arrays, QF_ABV (cf. Section 2.5.3). 


Then, the SMT formula for the unrolled transition system, prog , consists of the 
conjunction of all variable definitions, ^-functions, block conditions and control flow 
conditions. The postcondition defines the property that a basic block has to be 
visited. The visit of a basic block b will be forced by setting its visit flag to true, 
visit_b == 1. If b has been unrolled k times in a loop, the postcondition consists of 
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the disjunction of the visit flags of the corresponding basic blocks, postcondition = 
(Vi=o (visit_bi) == 1). The preconditions are optional and can be used for modelling 
assumptions; for instance, if variables of a function are known to be limited before 
they will be used in the modelled program semantics. Finally, the SMT formula <p 
will be defined as p = preconditions A prog A postcondition. 

If the SMT solver returns UNSAT for p, then there exists no solution for the input 
crafting problem for the unrolled transition system. The bound k can be increased 
and the procedure repeated. If p is satisfiable, then the SMT solver returns a model 
m. m consists of a set of assignments for all variables, visit flags and memory values. 
These depend on the control flow to the basic block b that must be visited. The 
inputs can be parsed in m by locating the variable or memory address where they 
will be defined and stored. Additionally, it is possible to reconstruct the execution 
path by filtering the true visit flags of the basic blocks. 



4 Discussion 


After the approach of bounded model checking for binary input crafting has been 
described, the following sections discuss its strengths and weaknesses as well as its 
applicability on real-world programs. For this, the methodology will be outlined, 
which forms the basis for the case studies that follow. Then, the experimental setup 
will be described. Following, different case studies will be performed, which facilitate 
an evaluation of this approach. Finally, based on the results of this chapter, the 
research question will be answered: Is bounded model checking for binary input 
crafting feasible and efficient? 


4.1 Methodology 

To examine the feasibility and efficiency of bounded model checking for binary input 
crafting, the characteristics of bounded model checking on the binary level must be 
evaluated. These characteristics will be assumed to be the impact of loops, function 
calls, API calls, memory operations, the amount of instructions and the number of 
paths. This assumption is premised on the fact that loops, function calls, API calls 
and memory operations are common structures on the assembly level. In addition, 
since the amount of instructions and the number of paths can be large, they may 
have a considerable influence on efficiency. 

To evaluate these characteristics, case studies will be designed that combine some of 
those properties. For each case study, a concrete test case will be implemented and 
Cylyx will be used in its assessment. After this, Cylyx will be applied to real-world 
binaries to estimate the viability of this approach in practice. It must be admitted 
that the test cases, in general, cannot be representative for binary input crafting on 
the basis of bounded model checking. A multitude of factors exist that may influence 
the results, for instance, compiler optimisations, incomplete disassembly, imprecise 
modelling of the program semantics or program structures, which are not known 
to be difficult for SMT solvers. Therefore, the test cases cannot be complete and 
cannot cover all possibilities. Despite the test cases not being fully representative, 
a methodology based on the test cases supports classifying the advantages and 
disadvantages of this approach. 

In the following case studies, test cases will be designed for loop unrolling, nested 
arrays, function calls and path explosion. Loop unrolling allows for analysis of the 
impact of the amount of instructions for SMT solving. Furthermore, the case studies 
for nested arrays and function inlining will be used to evaluate the effects of memory 
operations and API calls. Lastly, the case study for path explosion examines the 
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efficiency of path selection by SMT solvers and compares the results to the efficiency 
of symbolic execution. 

It will be evaluated to what extend this approach can be applied independent to 
architecture. Therefore, some case studies will be carried out on different architectures. 
Other case studies, which are designed for different goals (e.g., path selection and 
feasibility on real-world binaries), will only be evaluated for the x86_64 architecture, 
since Cylyx has mostly been tested for this platform. 

The amount of instructions, as well as the time needed to generate and solve the SMT 
formulas, will be measured and compared. If it is practicable, the SMT formulas will 
be solved by two independent SMT solvers in order to obtain less biased results. If a 
user input satisfies the SMT formula but not the program semantics, the user input 
will be crafted by symbolic execution to verify the results of Cylyx. 

In contrast to the designed test cases, which analyse the strengths and weaknesses of 
the described approach, the main focus of examination for the real-world programs 
remains on feasibility. It will be determined whether or not the approach is feasible 
in practice and which challenges may occur. 

In general, the structure of a test case takes the following form: A set of integers 
builds the input of a function. The function performs operations in dependence 
on the inputs and returns 1 if certain conditions are true, which depend on the 
input itself; otherwise, it returns 0. The function may contain loops or calls to 
other functions of a similar form, but no API calls. The crafted inputs satisfy the 
conditions such that the function returns 1. If not stated otherwise, the test cases 
will be compiled for the x86_86 architecture, as mentioned above. 


4.2 Experimental setup 

In this section, the experimental setup for the proceeding case studies will be 
described. The hardware configuration consists of an Intel Core i7-620M (2 cores, 
4 MB cache, 2.66 GHz), 8 GB RAM and 10 GB swap space. Based on an Arch 
Linux (80| , test cases will be compiled at optimisation level 0, with the GNU Compiler 
Collection (GCC) |81 (version 5.2.0) and debugged with the GNU Project Debugger 
(GDB) [82 . QEMU [83 and Miasm’s just-in-time compilation will be used to 
emulate executables of non-native targets. Symbolic execution will be performed on 
the basis of scripts, which are included in Cylyx, if operations of Cylyx have to be 
verified; to perform symbolic execution for path exploration, angr 1 24 , a framework 
for binary program analysis, will be used. The reason for choosing angr is twofold. 
First, it performs symbolic execution on the binary level for different architectures. 
Second, its ‘approach to symbolic execution draws on concepts proposed’ (p. 8 1 24 ) 
in KLEE jTj, FuzzBALL [23] and Mayhem [22 1, which are concepts of well-known 
symbolic execution engines, ‘adapted to [the authors’] specific problem domain’ (p. 
8 (24]). 
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Furthermore, in this setup, Cylyx relies on Miasm’s last git commit from 30th 
October 2015. To solve SMT formulas, the current versions of the SMT solvers, Z3 
(version 4.4.2) and Boolector |84j (version 2.1.1), will be utilised. The translation 
of Miasm IR into SMT formulas for Z3 justifies the usage of this SMT solver. In 
addition, Boolector will be used in comparison to Z3, since it won first place in the 
track QF ABV of the 10th International Satisfiability Modulo Theories Competition 
(SMT-COMP 2015) [85 ; as stated in Section 
included in this track. To solve such an SMT formula with Boolector, this formula 
will be transformed so that it is compatible with the SMT-LIB standard. This 
transformation is based on a feature of Z3. Lastly, for API function inlining, the 
library dietlibc J86] (version 0.33), which is optimised for small size, will be used, 
since the resulting assembly code is less complex. 


3.7, all formulas created by Cylyx are 


4.3 Case study 1: loop unrolling 

In this case study, the impact of the amount of SSA instructions on SMT solving 
will be examined. In addition, it will be demonstrated that SMT solvers are able to 
exploit structures of certain loops. 

4.3.1 Description 

The function consists of a loop that increments a counter up to a certain value. 
Within this loop, the truth value of a condition, which is based on the loop counter 
and the function’s input parameters, decides if the function returns 1. For instance, 
as illustrated in Figure 4.1, based on the input parameters a and b as well as 


the loop counter c, with 0 < c < 1000, the function foo returns 1 if and only if 
(a + b == 1337) A (c == 50) is true. In the following analysis, this loop has been 
unrolled with increasing unrolling bounds. 

4.3.2 Analysis 


Table [4~T| shows, in relation to Figure 4.1 the number of SSA instructions for each 
unrolling bound, as well as the measured time for SMT formula generation and 
solving. Since loop unrolling copies the basic blocks of the loop body in each unrolling 
step k times, as it can be seen, the number of the SSA instructions rises linearly to the 
unrolling bound. The creation of the SMT formula consists of the process described 
in Section |3.1.2| excluding SMT solving. Since the procedures of generating SMT 
formulas are identical up to the step of loop detection, the following steps determine 
the required time: loop unrolling, instruction rewriting, SSA transformation and 
translating the SSA instructions into SMT formulas. In other words, the generation of 
SMT formulas mainly depends on the polynomial complexity of the graph algorithms, 
at least in this case. 


50 


4.3. CASE STUDY 1: LOOP UNROLLING 


int foo(int a, int b) 

{ 

int c = 0; 

while (counter < 1000) 

{ 

if ((a + b == 1337) kk (c == 50)) 
return 1; 

counter ++ ; 

> 

return 0; 

> 

Figure 4.1: Simple loop condition relying on an incremented counter 



SSA instructions -10 5 

Figure 4.2: Polynomial growth in the number of SSA instructions 


Since the number of paths grows exponentially in the unrolling bound, it may be 
assumed that the time for SMT solving grows also exponentially in the unrolling 
bound or, equivalently, for this case, in the number of SSA instructions. However, as 


Table [44] and the corresponding visualisation in Figure 4.2 show, the SMT solving 
time grows polynomially in the number of SSA instructions. It can be proven by curve 
fitting (the construction of a mathematical function which has the best fit to a set of 
data tuples [87] ) that the curves of formula generation and Z3 grow quadratically and 
the curve of Boolector grows cubically. To put it differently, although the number 
of paths grows exponentially in the unrolling bound, SMT solving time remains 
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unroll 

factor 

# SSA 
instructions 

formula 

generation 

Z3 

Boolector 

0 

90 

0.8 s 

0.01 

s 

0.01 s 

5 

421 

1.3 s 

0.02 

s 

0.02 s 

10 

731 

2.0 s 

0.03 

s 

0.03 s 

20 

1351 

3.4 s 

0.04 

s 

0.07 s 

50 

3211 

8.0 s 

0.11 

s 

0.42 s 

100 

6311 

16.2 s 

0.25 

s 

1.28 s 

200 

12511 

34.4 s 

0.61 

s 

3.81 s 

500 

31111 

98.8 s 

3.90 

s 

19.89 s 

1000 

62111 

240.6 s 

20.40 

s 

82.38 s 

1500 

93111 

506.96 s 

45.51 

s 

182.02 s 

2000 

124111 

797.94 s 

80.75 

s 

329.94 s 

3000 

186111 

1395.97 s 

218.31 

s 

742.84 s 


Table 4.1: Influences of loop unrolling on SMT solvers 


polynomially. From this, it follows that SMT solvers exploit the high-level structure 
of this function. 

An SMT solver has to learn two conditions; a + b == 1337 and c == 50 must be 
true. Due to the SSA form, an SMT solver deduces that a and b are free variables, 
(a and b are free variables because they do not depend on previous calculations. In 
other words, they will be used on the right-hand side before they will be redefined 
on the left-hand side.) Therefore, guessing a solution is straightforward. For the 
second condition, if an SMT solver guesses a value 1000 > d > 50, it learns that the 
condition c < d must hold and will prune the search space for all paths that do not 
satisfy this condition. Last, if the SMT solver guesses d > 1000, it learns that the 
loop cannot be entered, since the loop condition 0 < d < 1000 does not hold. 

To conclude, while the number of SSA instructions grows linearly in the unrolling 
bound, the time for SMT solving grows polynomially. Therefore, SMT solvers are 
able to exploit the structure of this loop. Lastly, it must be noted that, on average 
in this case study, Z3 was four times as fast as Boolector. 


4.4 Case study 2: nested arrays 


The second case study was designed to examine the efficiency of memory operations, 
nested conditions and memory aliasing for different architectures. The test case 
includes function inlining and is loop-free. 
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4.4.1 Description 


In this case study, a function contains operations on arrays, whose values are initialised 
in dependence on the input parameters. Then, several conditions, based on the 
input parameters and the arrays, redefine certain array elements such that memory 
aliasing may occur. Additionally, a function of the same structure will be called, 
whose return value is part of the condition that decides if 1 will be returned. For 
instance, the implementation, which is the basis of the following analysis and is 
illustrated in Listing B.l| has two input parameters, a and b. A satisfying model is 
[a = 1431655878, b = 1431656874], 


4.4.2 Analysis 

Similar to the previous case study, Table |4.2| shows the number of SS A instructions 
and the times of generating and solving the SMT formulas. Instead of an unrolling 
bound, the first column represents the architecture for which the binary has been 
created - the main architectures supported by Miasm. 


architecture 

# SSA 
instructions 

formula 

generation 

Z3 

Boolector 

MIPS32 

234 

1.7 s 

49.1 s 

2.1 s 

ARMv7 

245 

2.4 s 

21.5 s 

0.4 s 

AARCH64 

268 

2.9 s 

162.3 s 

0.9 s 

x86 

412 

1.6 s 

12.8 s 

0.8 s 

x86_64 

402 

1.7 s 

31.7 s 

1.0 s 


Table 4.2: Influence of nested arrays on different architectures 


First, Table 4.2 shows that the SMT formula will be generated and solved for 
each architecture between a period of three to four seconds. However, a significant 
difference between Z3 and Boolector exists: Boolector is up to 162 times faster than 
Z3. Furthermore, the time it takes for Boolector to solve SMT formulas is mostly 
consistent across different architectures, while the times for Z3 are inconsistent. 


Without generalising, one possible reason for this might be the assumption that 
Boolector is much more efficient in memory operations (in the theory of arrays) 
than Z3, since calling functions and operations on arrays depend heavily on memory 
operations. This could be substantiated by results of the competition SMT-COMP 
2015, in which Boolector is seven times faster than Z3 in the main track of QFABV 
(sequential performance) [88 , all while solving more benchmarks. 

For the MIPS32 and ARMv7 architectures, the SMT solvers return a model for a 
and b, in which the inputs do not direct the control flow such that 1 will be returned. 
However, the execution path in the model is a possible execution path that may or 
may not be satisfiable. Therefore, the SMT formula encodes the control flow, but 
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does not represent the program semantics. To verify this result, the inputs will be 
crafted symbolically, based on Miasm’s intermediate representation. The derived 
path conditions will be solved by Z3, whereby a model contains crafted inputs. In 
this case, these inputs also do not have the desired characteristics. Thus, it is safe to 
assume that some transformations from assembly instructions into Miasm IR do not 
represent the instruction semantics. 


Lastly, while Z3 requires 31.7 seconds for 402 SSA instructions on the x86 64 

the SMT solver locates a solution in 20.4 seconds 
As a result, it can be said that no 


architecture, as seen in Table 4.2 


for the 62111 SSA instructions in Table 4.1 


necessary link exists between the amount of SSA instructions and the time it takes 
to solve an SMT formula. Therefore, it can be deduced that the efficiency of input 
crafting on the basis of bounded model checking depends on SMT solvers’ ability to 
exploit the underlying program structure. 


4.5 Case study 3: function inlining 

To improve the results of the preceding section, this case study will also be examined 
for different architectures and includes memory operations and nested conditions. 
The focus here is on complex code, nested function calls and inlining of API functions. 


4.5.1 Description 


The structure of this test case differs from the usual design. Firstly, a function foo 
may return different values, in dependence on the inputs. Then, it calls a function 
bar, which performs calculations on the inputs, but returns a constant value. A 
function main calls API functions and foo. Lastly, a condition in main depends on 
the return value of foo. 


In the code listed in Listing B.2 the function main calls the API function atoi 
thrice, once for each input a, b and c. If foo returns 4, then the API function printf 
will be called and will print a string. If foo returns a different value, then printf 
will print a different string. The function foo contains a loop, nested conditions 
and array operations. Additionally, it calls bar, which returns 4; foo will return 
4 if and only if bar will be called. A satisfying model is m = [a = 416029827, b = 
2055293730, c = 254662596]. 


Subsequently, two different scenarios will be examined. Firstly, to investigate the 
efficiency of complex code consisting of function calls, loops, arrays and arithmetic 
operations, the input will be crafted for foo such that bar is called and foo returns 
4. This will be examined for different architectures. Secondly, the efficiency of API 

call inlining will be inspected. For this, for the x86 64 architecture, the code will 

be crafted for main such that foo returns 4. The API function atoi will be inlined, 
while printf will be removed from the control flow graph. 
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4.5.2 Analysis 


As in the previous case study, Table 4.3 shows the number of SSA instructions, 
as well as the measured time of SMT formula generation and solving for different 
architectures. Additionally, it includes the starting point of the input crafting. The 
last row in the table represents the test case with API function inlining. Loops have 
been unrolled nine times. 


architecture 

# SSA 
instructions 

formula 

generation 

Z3 

Boolector 

start 

MIPS32 

681 

3.0 s 

38.2 s 

14.9 s 

f 00 

ARMv7 

875 

4.0 s 

> 7200 s 

> 7200 s 

f 00 

AArch64 

- 

- 

- 

- 

f 00 

x86 

1199 

3.6 s 

20.0 s 

32.0 s 

f 00 

x86_64 

1181 

3.9 s 

24.4 s 

11.8 s 

f 00 

x86_64 

4008 

9.4 s 

> 43200 s 

278.8 s 

main 


Table 4.3: Effects of function inlining on SMT solvers 


First, as it can be seen, no solutions exist for the AARCH64 and ARMv7 architectures. 
In the case of the former, some assembly instructions are not supported by Miasm. In 
the case of the latter, both SMT solvers do not terminate within two hours. Further, 
as in the previous case study, inputs crafted by symbolic execution do not satisfy 
the path condition. Therefore, it can be assumed that some transformations from 
assembly code into the intermediate representation do not represent the corresponding 
instruction semantics. These examples demonstrate the necessity to model program 
semantics precisely. 


Second, the requisite time to solve SMT formulas differs for all architectures. 

Boolector requires less time to solve the formula for the x86 64 architecture than 

for the x86 architecture, although the amount of SSA instructions as well as the 
underlying program structure, which the SMT solver exploits, are similar. A reason 
for this may be the usage of different calling conventions on the assembly level. 
In this test case, for the x86 architecture, function parameters are passed on the 
stack, while, for the x86_64 architecture, function parameters are passed in registers. 
To specify, since Cylyx utilises a global memory model (cf. Section 3.6), an SMT 
formula contains nested array writes. Deeply nested array writes are known to be 
less efficient |89 . Therefore, it may be the general case that, for bounded model 
checking, operations in the theory of arrays are less efficient than in the theory of bit 
vectors. 


Third, as evidenced by Table 4.3 SMT solving requires significantly more time when 
API functions are inlined. While Boolector is 23 times slower with API function 
inlining than without it, Z3 does not terminate within 12 hours. The reasons 
for this are twofold. On the one hand, the structure of the API functions and 
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their instructions build an additional overhead. On the other hand, function calls 
themselves may be less efficient, since a function operates in its own memory space 
and an SMT solver has to keep track of the links between the memory spaces, as 
well as the operations inside the function. Furthermore, this is supported by the 
assumption that operations in the theory of arrays may be less efficient. Last, it must 


be denoted that the results of this case study sustain the observation of Section 4.4 


which states that Z3 seems to be far less efficient with respect to memory operations 
than Boolector. 


To conclude, the results of this case study are that API call inlining is inefficient, 
while function inlining and, in particular, its corresponding memory operations 
may decrease the efficiency of binary bounded model checking. Additionally, the 
importance of preserving the program semantics and Boolector ’s advantage on 
memory operations have been pointed out. 


4.6 Case study 4: path selection 

In the following case study, the efficiency of SMT solvers on the path selection 
problem will be examined. Then, the results will be compared to the performance of 
a symbolic execution engine. 


4.6.1 Description 


To evaluate the efficiency of SMT solvers in path exploration, this case study was 
designed such that it is improbable that an SMT solver can exploit the program 
structure with less effort. The goal is to generate a number of execution paths that 
are, as far as possible, independent from each other. For this, opaque predicates will 
be utilised. Informally, an opaque predicate is a predicate of first-order logic, whose 
truth value is known a priori 1 90 . To put it differently, an opaque predicate can 
either only be true or only be false. 


The general structure of the function in this test study consists of n conditions, 
ci, ... ,c n , which depend on the user input and can be true or false. This function 
will return 1 if the conjunction of these n conditions is true. Then, a will be 
replaced by n new conditions, d\, . . . , d n , whereby each di consists of the disjunction 
of the corresponding c t and k distinct opaque predicates. Each opaque predicate 
evaluates as false and depends on the user input. In total, there exist n ■ k distinct 
opaque predicates. Figure T3 illustrates this process for n = 4. To exemplify, the 
function returns 1, if the conditions ci,...,C4 are true. Let, for instance, c\ be 
ci = (a + b == 1337), whereby a and b are the functions’ inputs. Then, ci will 
be replaced with d\ = ci V opi V • • • V opk, whereby all opt , with i e {1, . . . , k}, are 
pairwise distinct opaque predicates. After that, the function will return 1, if the 
conjunction of all di, d\ A • • • A d±, is true. As a result, the function returns true if 
and only if each c, is true, since the opaque predicates cannot be true (they evaluate 
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as false by design). In the control flow graph of this function, many paths exist such 
that 1 will be returned, but only one path is satisfiable. 

if ( ( ci || opi || ••• || opk ) 

&& (c 2 || op k+ 1 || ••• || 

&& (c 3 || op 2 k+i || ••• 

&& (c 4 || Op 3 k+ 1 II ••• 

return 1 ; 

Figure 4.3: Obfuscating a condition with opaque predicates 


OP2fc) 
OP3fc) 
opik ) ) 


In the implementation of this test case, the function has two input parameters 
and it holds n = 4, as illustrated in Figure 4.3 In the analysis, the bound k, 


representing the number of opaque predicates for each condition c*, will be increased 
and the input for this function crafted by an SMT solver, ci, c 2 , c 3 and c 4 are 
the conditions, which depend on the user input, a and b, and have to be true; 
Further, opi,...,op 4 k represent the 4 • k distinct opaque predicates, which also 
depend on the user input. For instance, an opaque predicate may be of the form 
(((a + 895543861) == (a + (a • a • 1294569136) + 1931398089))), for bit vector a of 
the length l = 32. 


The opaque predicates have been randomly generated and are proven to be unsatis- 
fiable by an SMT solver for bit vectors of the length l = 32, both in general and on 
the assembly level. To generate an opaque predicate, arbitrary arithmetic expressions 
(without bit shifts, division and modulo operations to exclude division by zero) will 
be randomly generated. Expressions depend on a bit vector variable a and random 
constants of the same size as a. If an SMT solver can prove, within two seconds, 
that the equality of two of those expressions is unsatisfiable, then it is an opaque 
predicate that evaluates to false. Limiting the decision time of the SMT solver to 
two seconds implies that a single opaque predicate cannot cause a long running time 
of an SMT solver, which would falsify the results. 


4.6.2 Analysis 


Table 4.4 shows the number of SSA instructions, as well as the time to generate 
and solve an SMT formula, for different numbers of opaque predicates k. In the 
following, k represents the total number of opaque predicates. Since n = 4, there 
exist | opaque predicates for each a. For instance, if k = 800, then each disjunction 
contains 200 opaque predicates. Additionally, the table lists the time in which angr, 
a symbolic execution framework, locates the one possible path. Path exploration 
with angr has been performed on the basis of Listing B.3| To recapitulate, angr will 

be used since it is able to perform symbolic execution on the x86 64 architecture 

and since it is based on techniques, amongst others, proposed in KLEE [7|. 
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# predicates 

# SSA 
instructions 

formula 

generation 

Z3 

Boolector 

angr 

0 

147 

0.9 s 

0.0 

s 

0.0 

s 

2.6 s 

40 

2172 

7.0 s 

0.4 

s 

2.7 

s 

49.6 s 

80 

4074 

13.6 s 

1.1 

s 

6.2 

s 

110.2 s 

120 

5438 

18.2 s 

2.5 

s 

7.1 

s 

147.2 s 

200 

8270 

27.8 s 

5.1 

s 

6.4 

s 

329.5 s 

400 

16617 

57.2 s 

18.1 

s 

23.1 

s 

745.1 s 

800 

34028 

119.9 s 

111.5 

s 

18.1 

s 

1785.1 s 

1200 

51267 

180.0 s 

335.7 

s 

49.1 

s 

4265.0 s 

1600 

67107 

247.0 s 

715.9 

s 

208.3 

s 

5871.1 s 

2000 

82291 

325.1 s 

956.9 

s 

299.6 

s 

10534.5 s 


Table 4.4: SMT solvers and path exploration 


In theory, for a given k, there exist (| + l) 4 different paths between the function 
start and the basic block that returns 1. However, in practice, the total number of 
paths may differ for two reasons. First, the compiler may eliminate some opaque 
predicates during compilation. Second, each opaque predicate may be represented 
by several basic blocks in the intermediate representation. Thus, to find the best fit 
for the growth of the number of paths, this number will be calculated on the basis of 
the directed acyclic graph for some k. 14641 paths exist for k = 40 (47 basic blocks) 
and 176400 paths for k = 80 (85 basic blocks). In addition, there exist 4753392 
paths for k = 200 (190 basic blocks) and 77995008 paths for k = 400 (379 basic 
blocks). For k > 400, the calculation of the number of paths does not terminate after 
several hours. However, for k = 1200, the number of basic blocks is 1130 and 1837 
for k = 2000. Therefore, the number of paths grows polynomially, since the best fit 
for tuples consisting of the number of opaque predicates and the number of paths is 
a curve 3 or 4 degrees. 


As it can be seen in Table 4.4, SMT solvers are very efficient in path selection. For 
instance, Cylyx locates the only satisfiable path out of 77995008 possible paths 
( k = 400) within 18 seconds, using Z3 as the SMT solver. In this case, it is 10 times 
faster than angr, which finds the same path in 745 seconds. Further, for k = 2000, 
Cylyx is 17 times faster, using Boolector as the SMT solver. This is also illustrated 
in Figure |4~4| which visualises the results of both Cylyx and angr using the time for 
SMT formula generation and the minimal time for SMT solving. As a result, the 
factor of speed in which Cylyx is faster than angr grows in the number of opaque 
predicates. 


Since deciding the quantifier-free fragments of the theories of arrays and bit vectors 
is NP-complete (cf. Section 2.5.3), it will be expected that SMT solving becomes 
inefficient for some k. However, this k has not been found. For instance, Cylyx locates 
the satisfying path for k = 4000 (3679 basic blocks) in 1498.1 seconds, whereby 
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Boolector decides the SMT formula within 712.6 seconds; Cylyx is seven times faster 
for k = 4000 than angr is for k = 2000. 

Lastly, Z3 and Boolector differ in their timings. In the beginning, Z3 is faster than 
Boolector. However, for k > 800, Boolector is significantly faster. Since the previous 
results do not indicate a possible reason for that, it will be grasped as an observation. 


• 10 4 



Figure 4.4: Growth of path exploration in the number of opaque predicates 


4.7 Case study 5: real-world programs 

After the strengths and weaknesses of bounded model checking for binary input craft- 
ing have been examined, this case study investigates the applicability of this approach 
to real-world programs. In addition, possible challenges will be demonstrated. 

4.7.1 Description 

Subsequently, the process of input crafting on real-world binaries will be described. 
For this process, Cylyx will be utilised to analyse functions in two different binaries. 
Functions similar to the previous test cases will be chosen; functions may contain 
function calls, loops and receive inputs as function parameters. Those functions 
facilitate a comparison with the previous case studies, since they share similar 
characteristics. 

First, the function base64_decode_ctx of the binary base64 from the Coreutils |9l] 
will be examined. Second, input will be crafted for the function hwaddr_aton of the 
WPA supplicant [92]. Last, the results will be analysed. 


4.7. CASE STUDY 5: REAL-WORLD PROGRAMS 
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4.7.2 Base64 

The inputs of the function base64_decode_ctx are a context object, a pointer 
to an input buffer, a pointer to an output buffer and the length of both buffers. 
The input buffer contains a string, which is base64 encoded; the output buffer 
shall contain the decoded string, after the function returns. The function proto- 
type is [. . .] (struct base64_decode_context *ctx, const char ^restrict 
in, size_t inlen, char ^restrict out, size_t *outlen). This function con- 
sists of several loops, nested conditions, an API call to memchr and a function call to 
decode_4, which contains a loop and conditions. 

Cylyx is able to inline the functions and unroll the loops. However, it is not possible 
to inline the API function, since Miasm does not support transforming the assembly 
instruction pshufd into Miasm IR. As a result, the program semantics cannot be 
encoded precisely and input crafting cannot be performed. 


4.7.3 WPA supplicant 


The function hwaddr_aton converts an ASCII string into a hardware address, rep- 
resented as bytes. For instance, the hardware address ‘aa:bb:cc:dd:ee:ff’ will be 
converted into 0 xaabbccddeef f . Two parameters will be passed to this function - a 
pointer to a char buffer containing the ASCII string, in RDI, and a pointer into a 
buffer consisting of bytes, in RSI. The function calls another function, hwaddr_parse, 
and in dependence upon the return value of the called function, it returns 0 if the 
string is a valid hardware address; otherwise, it returns —1. Cylyx will be utilised 
to craft a valid input such that 0 will be returned; a valid hardware address will 
be crafted. However, the structure of the underlying functions will be discussed 
beforehand. The assembly code of those functions is listed in Appendix B.4| 


The called function, hwaddr_parse, consists of nested conditions, a loop and a 
function call to hex2byte within this loop. The loop will be passed a maximum of 
six times, whereby the ASCII string will be separated at the colon. For instance, the 
first loop iteration processes ‘aa’, the second loop iteration ‘bb’ and the last loop 
iteration ‘ff’. Each time, hex2byte will be called, which transforms a string into a 
byte; for instance, ‘aa’ will be transformed into Oxaa. This function contains nested 
conditions and two function calls to hex2num. To hex2num, a single character will be 
passed and the corresponding byte value returned. For instance, for the character ‘a’ 
with the ASCII value 0x61, the value Oxa will be returned. In total, to craft a valid 
hardware address, the loop has to be unrolled six times and 19 functions must be 
inlined. 


Cylyx generates an SMT formula with an unrolling bound of k = 7 in 10.2 seconds. 
In total, this SMT formula contains 4230 translated SSA instructions. Boolector 
locates a satisfying model within 258 seconds. However, the crafted input does not 
represent a valid hardware address, but the execution path is equal to that of a 
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dynamic trace, whose input is a valid hardware address. Further analysis reveals 
the reason for this behaviour. The SMT formula contains four free variables - RDI, 
RSI, RBP and RSP. (They are free because they are not bounded by other variables.) 
Therefore, the SMT solver has to estimate a satisfying memory layout, as well as 
values for those free variables. In this case, the SMT solver sets RDI and RSP to the 
same value. Since RSP operates as the stack pointer, which will be used for referencing 
local variables, and since RDI contains a pointer to the char array, this will cause a 
conflict, because two distinct memory regions overlap. To put it differently, although 
the SMT formula represents the program semantics of the function hwaddr_aton, 
the SMT solver assigns values for free variables such that the memory layout does 
not represent the program semantics. 

To solve this problem, preconditions will be applied. The preconditions set RDI, RSI, 
RSP and RBP to different values such that they cannot overlap. Then, Boolector solves 
the SMT formula in about 260 seconds. In this case, the crafted input represents a 
valid hardware address, ‘eA:79:82:f4:d9:ff’; the input can be crafted successfully. 


4.7.4 Analysis 

The results of this case study are threefold. Firstly, bounded model checking for 
binary input crafting is limited in practice, since the program semantics have to be 
precisely modelled. Both an API function, which cannot be inlined, or an assembly 
instruction, which cannot be transformed into the intermediate representation, may 
influence the approach. Second, SMT solvers may operate in an unexpected and 
unpredictable manner. If this behaviour is known or can be figured out, it may be 
handled by applying preconditions. Last, it has been shown that functions in real- 
world binaries exist, for which bounded model checking for binary input crafting can 
be applied efficiently. The underlying algorithms have not to be reverse engineered 
in detail; defining a starting point, a basic block that must be reached and a bound 
for loop unrolling are sufficient for crafting satisfying inputs. 


4.8 Conclusion 

As stated in Section [L3j the research question asks whether bounded model checking 
to craft inputs for binary programs is both feasible and efficient. To answer this 
question, a process to apply bounded model checking to binary programs has been 
described in Section [3j The main idea of this process is to create an SMT formula 
of a directed acyclic graph, which will be transformed into SSA. Then, in this 
chapter, methods have been derived and case studies designed such that the main 
characteristics of this approach, which are described in Section |4.1| can be examined. 
The Cylyx framework, which is based on methods described in Chapter [3j has been 
used for investigation. 


4.8. CONCLUSION 
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As exhibited in the previous sections, the number of SSA instructions is not related to 
the required time for SMT solving. In some cases, the satisfiability of 268 instructions 
in SSA form can be decided in several minutes by an SMT solver (cf. Section 4.4), 
while, in other cases, the satisfiability of 31111 SSA instructions can be decided 
within seconds (cf. Section 4.3). 

In the case of path selection, it has been shown that bounded model checking is very 


efficient, in many cases. For instance, in the case of loop unrolling in Section 4.3 


where the number of paths grows exponentially in the unrolling bound, bounded 
model checking remains polynomially since SMT solvers are able to exploit the 
structure of the program. Additionally, Section 4.6 demonstrated that, in other cases, 
bounded model checking is significantly more efficient in path selection than the 
symbolic execution engine angr. 

Furthermore, it has been shown that function inlining is less efficient. For this, a 
relation to the supposed inefficiency of memory operations (in particular, nested 
memory writes) has been assumed in Section 4.5 Primarily, it has been stated that 
inlining of API functions is inefficient. However, API functions may be handled more 


efficiently by modelling their behaviour, as described in Section 3.2.2 


Apart from these results, some observations have been made. First, Z3 seems to 
be significantly slower than Boolector in memory operations. Secondly, in general, 
predicting the necessary time to solve an SMT formula is difficult. In some cases, 
within the scope of this dissertation, it cannot be clearly stated whether SMT solvers 
are efficient or inefficient (cf. Section 4.6). Lastly, as denoted in Section 4.7.3 SMT 
solvers may operate differently than expected, which may be caused by underlying 
assumptions that have not been modelled explicitly. 


To summarise, bounded model checking is feasible for binary input crafting, as 
long as the program semantics can be precisely modelled. In practice, this precise 
modelling is the main limitation of this approach. In many cases, the approach works 
efficiently, since SMT solvers exploit the structure of the program semantics. As 
a result, they are very efficient in path selection. On the contrary, nested memory 
writes decelerate SMT solving. In general, this approach is limited by complexity, 
because it operates on problems that are, in the best case scenario, NP-complete. 
Nevertheless, it has been demonstrated that bounded model checking on binary 
programs is architecture-independent and can be applied efficiently and successfully 
to input crafting on real-world binaries. 



5 Future work 


This dissertation introduced bounded model checking for binary programs and 
its application to input crafting. In this chapter, future work will be discussed. 
Improvements and optimisations on the implementation level will be outlined and 
future research will be described. 


5.1 Implementation level 


In the following, concepts on the implementation level will be presented, which 
may improve the described process of bounded model checking on the binary level 
(cf. Chapter [3]). While some concepts may improve the precision of modelling the 
program semantics, other concepts may optimise its efficiency. 

Section [2. 1.6| illustrated the power of SSA to optimise code; SSA simplifies essentially 
algorithms for optimisations, such as dead-code elimination or constant propagation, 
data flow analysis and decompilation [93 . Therefore, in the process of bounded 
model checking, applying optimisations on the SSA instructions may significantly 
reduce the complexity of the SMT formula and improve the performance of the SMT 
solver. 


To enhance the modelling of program semantics, API functions have to be handled. 
Although it has been shown that API call inlining is inefficient, the area of application 
can be extended by preparing common API functions for inlining; this is especially 
true, if inlining is the only way to model the semantics of an API function. As 
mentioned in Section 3.2.2 SMT-based modelling may be a more efficient and 
architecture-independent method of handling API functions. The modelling of 
malloc and free is of special interest because it allows for the modelling of dynamic 
memory allocations. Sinz, Falke and Merz [38 demonstrated this in their LLBMC. 

In relation to loop unrolling, loop detection will be performed on the basis of strongly 
connected components, as described in Section 3.3.2[ Therefore, the loop structures 
of high-level programming languages will not be considered because only outermost 
loops will be detected. To consider the high-level structure and loop nesting, loop 
nesting trees |42| can be utilised for loop unrolling; nested loops will be unrolled 
iteratively, starting from the innermost loop and ending with the outermost loop. 
As a result, the SMT solver may exploit the loop structure more efficiently. 
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5.2 Future research 


In this dissertation, input crafting has been applied to bounded model checking 
on the binary level as a postcondition. However, arbitrary postconditions can be 
applied to an SMT formula that represents the logical encoding of a binary function. 
Building upon the strengths of SMT solvers, bounded model checking may operate 
powerfully in other areas related to binary application security. 


Section 4.6 demonstrated that SMT solvers are very efficient in path selection on 
the basis of opaque predicates, a common technique in the context of program 
obfuscation [90 . In addition, SMT solvers have been used to synthesise programs 1 94 
as well as for equivalence checking [12|. Therefore, bounded model checking on 
the binary level may be the groundwork for SMT-based obfuscation, deobfuscation, 
equivalence checking and program synthesis. 


Sinz, Falke and Merz 1 38 introduced bounded model checking on the basis of LLVM’s 
intermediate representation in order to locate integer overflows, illegal memory access 
and other vulnerabilities in C programs. This approach may be applied to the binary 
level. Furthermore, since arbitrary conditions can be encoded as SMT formulas, it is 
possible to define the values of arbitrary memory locations for a specific program 
state. For instance, this may be used to craft inputs that write malicious code into 
defined memory areas. Therefore, the approach of vulnerability discovery may be 
extended to automatic exploit generation on the basis of bounded model checking. 


A Contributions to Miasm 


Date 

Git commit 

Description 

24.04.2015 

4ff91550b953a661abaa49b936ae76a6b955df9f 

Submitted bugs in the Z3 translation. 

25.03.2015 

f72d0de75c6815db54f9d54824e8948d962eee5a 

Z3 translation has been added for idiv. 

25.03.2015 

31b70a4a6dc770bb21db97cdac27f7fle54055d7 

The x86 semantics have been added for 
pxor. 

01.05.2015 

f9bf6fbbce0d984549fbllel6cb9acfc7be00c86 

Redundant code has been removed and 
typography has been improved. 

03.06.2015 

eb85ae0dbll5b40190446c0aa841e8alb562eece 

Graph algorithms have been added to 
compute immediate dominators and 
the dominance frontier. 

06.07.2015 

35852d91259bcac6fl5c5db4edf2ccf50f00f444 

Submitted a sandbox class for x86 64 

Linux. 

29.07.2015 

8 1 7fc666eac74c802d4d592f50a3872a3197f4a5 

Z3 translations have been added for 
udiv, imod and umod. 

12.08.2015 

4d93231e77a8af094cl98a8b3b3733b0250cd45a 

Fixed a link in the documentation. 

06.09.2015 

Cc20f3d79c641d929865fl2471a70a2575b8cf54 

Submitted graph walks, natural loop 
detection and the computation of 
strongly connected components. 

16.10.2015 

861e0dc047b3a6675aa8a9bl31a53cb6d4dd033f 

Z3 translations have been added for 
left/right rotations. 


Table A.l: Contributions to Miasm, sorted by date of git merges 





B Listings 

B.l Case study 2 


1 int bar(int a, int b) 

2 { 

3 int ar [10] ; 

4 

5 ar[0] = 3; 

6 ar[l] = a; 

7 ar[2] = 2 * a + b; 

8 ar [3] = b << 3; 

9 ar[4] = b + a; 

10 ar [5] = 55 * a; 

11 ar [6] = ar [5] ; 

12 ar[7] = a + 1337; 

13 ar [8] = ar [0] + ar [4] ; 

14 ar[9] = 999; 

15 

16 if (ar[0] + 2 * ar[4] + ar[3] >> ar[0] 

17 { 

18 ar[ar[0] +2] =21; 

19 ar[3] = 11; 

20 ar [9] = a * ar [2] ; 

21 > 

22 

23 else 

24 { 

25 ar[3] = ar[l] + ar[0] ; 

26 ar[7] = 11; 

27 > 

28 

29 if (ar[9] == ar[7] + ar[3] + 123) 

30 return ar [9] + ar [3] ; 

31 else 

32 return ar [9] - ar [3] ; 

33 } 

34 

35 int f oo (int a, int b) 

36 { 

37 int ar [3] ; 

38 

39 ar [0] = 3; 

40 ar[l] = a; 

41 ar[2] = 2 * a + b; 

42 

43 

44 if (ar [0] + ar [2] == 1337) 

45 { 

46 ar [0] = 1 

47 ar[l] = 2 

48 ar[2] = a 

49 > 

50 

51 else 
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52 

53 

54 

55 

56 

57 

58 

59 

60 


{ 

ar[l] = ar[0] + ar[l]; 
ar[0] = ar[l] * b; 

> 

if (ar[0] + b + ar[l] == 1337 + bar (a + b , ar[0])) 
return 1 ; 

return 0; 


Listing B.l: Nested array operations and a function call 


B.2 Case study 3 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

44 


#include<stdlib . h> 

#include<stdio . h> 

int bar(int a, int b) ; 

int f oo (int a, int b, int c) 

{ 

int ar [3] ; 

ar[0] = a + b + c; 

ar[l] = a * b; 

ar[2] = ar[0] + c * ar[l] - (b * b) + 1; 
int d = (a * b) << c; 

if (ar[2] + ar[l] * c + a == 1337 + d) 

{ 

a = b; 
b = a * c; 
c = (a + b) * d; 
if (a + b == 200) 
return 2; 

else if (a + b) 

{ 

c = c + 1; 
d = b + 2; 

if (b+d+a==c*b+ 300) 

{ 

int i = 0; 
while (i < 4) 

{ 

i++; 

if ( (i + c > 100) && (i > 2)) 
return 3; 

> 

if (ar [2] + ar [1] ) 

{ 

if ((a < b) II (c > d)) 
return bar(a, b) ; 
return 5; 

> 

else 

{ 


if (a < b) 
return 6; 


B.3. CASE STUDY 4 


45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 

69 

70 

71 

72 

73 

74 

75 

76 

77 

78 

79 

80 
81 
82 


return 7 ; 

> 

> 

> 

return 1; 

> 

return 0; 


int bar(int a, int b) 

{ 

int d = a * b; 
a = b * b; 
d = 2; 
b = d * 2; 

return b; 

> 

int main (int argc, char * argv[]) 

int a,b,c; 

a = atoi(argv[l] ) ; 
b = atoi(argv[2] ) ; 
c = atoi(argv[3] ) ; 

if (foo(a, b, c) == 4) 

{ 

printf ("Correct ! \n") ; 
return 0; 

> 

else 

{ 

printf ( "Wrong ! \n" ) ; 
return -1; 

} 


Listing B.2: Nested conditions with a loop and a function call 


B.3 Case study 4 


1 

2 

3 

4 

5 

6 
7 


import sys 
import angr 
import simuvex 

if len(sys . argv) != 4: 

print " [*] Syntax: <filename> <start address> <end address>" 
sys . exit (0) 


9 

10 

11 

12 

13 

14 

15 


# sample 
f = sys. argv [1] 

start_addr = int (sys . argv [2] , 16) 
end_addr = int (sys . argv [3] , 16) 

b = angr. Pro ject(f, load_options={"auto_load_libs" : False}) 
initial_state = b. factory .blank_state (addr=start_addr , 
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16 remove_options={simuvex . o . LAZY_S0LVES}) 

17 

18 pg = b. factory .path_group(initial_state , immutable=False) 

19 pg. explore (f ind=end_addr) 

20 

21 if pg. found: 

22 found_state = pg. found [0] .state 

23 print "A satisfying path has been found." 

Listing B.3: Path selection with angr 


B.4 Case study 5 


1 

0x00415f3c push rbp 

2 

0x00415f3d mov rbp, rsp 

3 

0x00415f40 sub rsp, 0x10 

4 

0x00415f44 mov qword [rbp - 8] , rdi 

5 

0x00415f48 mov qword [rbp - 0x10] , rsi 

6 

0x00415f4c mov rdx, qword [rbp - 0x10] 

7 

0x00415f50 mov rax, qword [rbp - 8] 

8 

0x00415f54 mov rsi, rdx 

9 

0x00415f57 mov rdi, rax 

10 

0x00415f5a call sym.hwaddr_parse 

11 

0x00415f5f test rax, rax 

12 

0x00415f 62 je 0x415f6b 

13 

0x00415f64 mov eax, 0 

14 

0x00415f69 jmp 0x415f70 

15 

0x00415f6b mov eax, Oxffffffff 

16 

0x00415f70 leave 

17 

0x00415f71 ret 


Listing B.4: WPA supplicant: hwaddr_aton 


1 0x00415ebe push rbp 

2 0x00415ebf mov rbp, rsp 

3 0x00415ec2 sub rsp, 0x20 

4 0x00415ec6 mov qword [rbp - 0x18] , rdi 

5 0x00415eca mov qword [rbp - 0x20] , rsi 

6 0x00415ece mov qword [rbp - 8] , 0 

7 0x00415ed6 jmp 0x415f2f 

8 0x00415ed8 mov rax, qword [rbp - 0x18] 

9 0x00415edc mov rdi, rax 

10 0x00415edf call sym.hex2byte 

11 0x00415ee4 mov dword [rbp - Oxc] , eax 

12 0x00415ee7 cmp dword [rbp - Oxc] , 0 

13 0x00415eeb jns 0x415ef4 

14 0x00415eed mov eax, 0 

15 0x00415ef2 jmp 0x415f3a 

16 0x00415ef4 add qword [rbp - 0x18] , 2 

17 0x00415ef9 mov rdx, qword [rbp - 0x20] 

18 0x00415efd mov rax, qword [rbp - 8] 

19 0x00415f01 add rax, rdx 

20 0x00415f04 mov edx, dword [rbp - Oxc] 

21 0x00415f07 mov byte [rax] , dl 

22 0x00415f 09 cmp qword [rbp - 8] , 4 

23 0x00415f0e ja 0x415f2a 

24 0x00415fl0 mov rax, qword [rbp - 0x18] 
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25 0x00415fl4 lea rdx, qword [rax + 1] 

26 0x00415f 18 mov qword [rbp - 0x18] , rdx 

27 0x00415flc movzx eax, byte [rax] 

28 0x00415flf cmp al , 0x3a 

29 0x00415f 21 je 0x415f2a 

30 0x00415f23 mov eax, 0 

31 0x00415f28 jmp 0x415f3a 

32 0x00415f2a add qword [rbp - 8] , 1 

33 0x00415f2f cmp qword [rbp - 8] , 5 

34 0x00415f 34 jbe 0x415ed8 

35 0x00415f36 mov rax, qword [rbp - 0x18] 

36 0x00415f3a leave 

37 0x00415f3b ret 

Listing B.5: WPA supplicant: hwaddr_parse 


1 

0x00415e55 push rbp 

2 

0x00415e56 mov rbp, rsp 

3 

0x00415e59 sub rsp, 0x18 

4 

0x00415e5d mov qword [rbp - 0x18] , rdi 

5 

0x00415e61 mov rax, qword [rbp - 0x18] 

6 

0x00415e65 lea rdx, qword [rax + 1] 

7 

0x00415e69 mov qword [rbp - 0x18] , rdx 

8 

0x00415e6d movzx eax, byte [rax] 

9 

0x00415e70 movsx eax, al 

10 

0x00415e73 mov edi, eax 

11 

0x00415e75 call sym.hex2num 

12 

0x00415e7a mov dword [rbp - 4] , eax 

13 

0x00415e7d cmp dword [rbp - 4] , 0 

14 

0x00415e81 jns 0x415e8a 

15 

0x00415e83 mov eax, Oxffffffff 

16 

0x00415e88 jmp 0x415ebc 

17 

0x00415e8a mov rax, qword [rbp - 0x18] 

18 

0x00415e8e lea rdx, qword [rax + 1] 

19 

0x00415e92 mov qword [rbp - 0x18] , rdx 

20 

0x00415e96 movzx eax, byte [rax] 

21 

0x00415e99 movsx eax, al 

22 

0x00415e9c mov edi, eax 

23 

0x00415e9e call sym.hex2num 

24 

0x00415ea3 mov dword [rbp - 8] , eax 

25 

0x00415ea6 cmp dword [rbp - 8] , 0 

26 

0x00415eaa jns 0x415eb3 

27 

0x00415eac mov eax, Oxffffffff 

28 

0x00415ebl jmp 0x415ebc 

29 

0x00415eb3 mov eax, dword [rbp - 4] 

30 

0x00415eb6 shl eax, 4 

31 

0x00415eb9 or eax, dword [rbp - 8] 

32 

0x00415ebc leave 

33 

0x00415ebd ret 


Listing B.6: WPA supplicant: hex2byte 


1 0x00415e06 push rbp 

2 0x00415e07 mov rbp, rsp 

3 0x00415e0a mov eax, edi 

4 0x00415e0c mov byte [rbp - 4] , al 

5 0x00415e0f cmp byte [rbp - 4] , 0x2f 

6 0x00415el3 jle 0x415e24 

7 0x00415el5 cmp byte [rbp - 4] , 0x39 

8 0x00415el9 jg 0x415e24 

9 0x00415elb movsx eax, byte [rbp - 4] 
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10 0x00415elf sub eax, 0x30 

11 0x00415e22 jmp 0x415e53 

12 0x00415e24 cmp byte [rbp - 4] , 0x60 

13 0x00415e28 jle 0x415e39 

14 0x00415e2a cmp byte [rbp - 4] , 0x66 

15 0x00415e2e jg 0x415e39 

16 0x00415e30 movsx eax, byte [rbp - 4] 

17 0x00415e34 sub eax, 0x57 

18 0x00415e37 jmp 0x415e53 

19 0x00415e39 cmp byte [rbp - 4] , 0x40 

20 0x00415e3d jle 0x415e4e 

21 0x00415e3f cmp byte [rbp - 4] , 0x46 

22 0x00415e43 jg 0x415e4e 

23 0x00415e45 movsx eax, byte [rbp - 4] 

24 0x00415e49 sub eax, 0x37 

25 0x00415e4c jmp 0x415e53 

26 0x00415e4e mov eax, Oxffffffff 

27 0x00415e53 pop rbp 

28 0x00415e54 ret 

Listing B.7: WPA supplicant: hex2num 
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